[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:28:28 dist-test-slave-gp6n krb5kdc[30018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:28:28 dist-test-slave-gp6n krb5kdc[30018](info): set up 2 sockets
Aug 08 18:28:28 dist-test-slave-gp6n krb5kdc[30018](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260808 18:28:29.960728 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.005s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:28:30 dist-test-slave-gp6n krb5kdc[30018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:28:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:28:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:28:30.097834 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:37375
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:37375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:30.222649 30032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:30.223045 30032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:30.223140 30032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:30.229404 30032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:28:30.229496 30032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:30.229562 30032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:30.229602 30032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:28:30.229640 30032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:28:30.236069 30032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:37375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:37375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:30.237360 30032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:30.238718 30032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:30.248997 30037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:30.249050 30038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:30.249073 30032 server_base.cc:1034] running on GCE node
W20260808 18:28:30.249015 30041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:30.249850 30032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:30.251343 30032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:30.252537 30032 hybrid_clock.cc:648] HybridClock initialized: now 1786213710252528 us; error 50 us; skew 500 ppm
Aug 08 18:28:30 dist-test-slave-gp6n krb5kdc[30018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213710, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:30.255482 30032 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:28:30.256364 30032 webserver.cc:460] Webserver started at http://127.29.78.190:39243/ using document root <none> and password file <none>
I20260808 18:28:30.256880 30032 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:30.256939 30032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:30.257184 30032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:30.259240 30032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "963d27d23cc149de8f179b5ef233cd03"
format_stamp: "Formatted at 2026-08-08 18:28:30 on dist-test-slave-gp6n"
server_key: "4fa409cf52d341d5d0fd99c83d476b4a"
server_key_iv: "c2547b2916243cfef32b951fef6b837a"
server_key_version: "encryptionkey@0"
I20260808 18:28:30.259747 30032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "963d27d23cc149de8f179b5ef233cd03"
format_stamp: "Formatted at 2026-08-08 18:28:30 on dist-test-slave-gp6n"
server_key: "4fa409cf52d341d5d0fd99c83d476b4a"
server_key_iv: "c2547b2916243cfef32b951fef6b837a"
server_key_version: "encryptionkey@0"
I20260808 18:28:30.263543 30032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 18:28:30.265856 30048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:30.267117 30032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:28:30.267238 30032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "963d27d23cc149de8f179b5ef233cd03"
format_stamp: "Formatted at 2026-08-08 18:28:30 on dist-test-slave-gp6n"
server_key: "4fa409cf52d341d5d0fd99c83d476b4a"
server_key_iv: "c2547b2916243cfef32b951fef6b837a"
server_key_version: "encryptionkey@0"
I20260808 18:28:30.267336 30032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:30.280320 30032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:30.283473 30032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:30.283718 30032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:30.292091 30032 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:37375
I20260808 18:28:30.292114 30108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:37375 every 8 connection(s)
I20260808 18:28:30.293177 30032 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:28:30.295488 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 30032
I20260808 18:28:30.295643 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:28:30.296979 30010 external_mini_cluster.cc:1428] Setting key 658e23e578f96bfffad7b3e2176d4160
I20260808 18:28:30.297159 30109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:30.303869 30109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03: Bootstrap starting.
I20260808 18:28:30.307322 30109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:30.308919 30109 log.cc:824] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:30.311626 30109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03: No bootstrap required, opened a new log
I20260808 18:28:30.316665 30109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963d27d23cc149de8f179b5ef233cd03" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37375 } }
I20260808 18:28:30.316959 30109 raft_consensus.cc:374] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:30.317020 30109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 963d27d23cc149de8f179b5ef233cd03, State: Initialized, Role: FOLLOWER
I20260808 18:28:30.317597 30109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963d27d23cc149de8f179b5ef233cd03" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37375 } }
I20260808 18:28:30.317806 30109 raft_consensus.cc:388] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:28:30.317885 30109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:28:30.317982 30109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03 [term 0 FOLLOWER]: Advancing to term 1
Aug 08 18:28:30 dist-test-slave-gp6n krb5kdc[30018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:30.319082 30109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963d27d23cc149de8f179b5ef233cd03" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37375 } }
I20260808 18:28:30.319447 30109 leader_election.cc:302] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 963d27d23cc149de8f179b5ef233cd03; no voters:
I20260808 18:28:30.319770 30109 leader_election.cc:288] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:28:30.319916 30114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:30.320119 30114 raft_consensus.cc:686] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03 [term 1 LEADER]: Becoming Leader. State: Replica: 963d27d23cc149de8f179b5ef233cd03, State: Running, Role: LEADER
I20260808 18:28:30.320442 30114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963d27d23cc149de8f179b5ef233cd03" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37375 } }
I20260808 18:28:30.321149 30109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:28:30.322728 30116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "963d27d23cc149de8f179b5ef233cd03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963d27d23cc149de8f179b5ef233cd03" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37375 } } }
I20260808 18:28:30.322860 30116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03 [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:30.323230 30115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 963d27d23cc149de8f179b5ef233cd03. Latest consensus state: current_term: 1 leader_uuid: "963d27d23cc149de8f179b5ef233cd03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963d27d23cc149de8f179b5ef233cd03" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37375 } } }
I20260808 18:28:30.323407 30115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03 [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:30.323813 30124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:28:30.327356 30111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:30.300919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39470 (local address 127.29.78.190:37375)
0808 18:28:30.301456 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:30.301475 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:30.302493 (+ 1018us) server_negotiation.cc:407] Connection header received
0808 18:28:30.307023 (+ 4530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:30.307042 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:30.307339 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:30.307669 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:30.311172 (+ 3503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:30.312390 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:30.314049 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:30.314747 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:30.314865 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:30.319912 (+ 5047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:30.319918 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:30.319929 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:30.319954 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:30.322855 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:30.323796 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:30.323803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:30.323812 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:30.323977 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:30.324631 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:30.324635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:30.324637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:30.325228 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:30.325428 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:30.326058 (+ 630us) server_negotiation.cc:299] Negotiation successful
0808 18:28:30.326327 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":150,"threads_started":1}
I20260808 18:28:30.327729 30124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:28:30.334008 30124 catalog_manager.cc:1347] Generated new cluster ID: 588704fa92cc436887b2f976f32c8301
I20260808 18:28:30.334077 30124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:28:30.351290 30124 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:28:30.352253 30124 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:28:30.365216 30124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 963d27d23cc149de8f179b5ef233cd03: Generated new TSK 0
I20260808 18:28:30.365955 30124 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260808 18:28:30.406208 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:37375
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:30.517021 30136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:30.517297 30136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:30.517376 30136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:30.521370 30136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:30.521458 30136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:30.521544 30136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:28:30.527019 30136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:37375
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:30.528082 30136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:30.529124 30136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:30.539633 30144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:30.539625 30141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:30.539629 30142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:30.539819 30136 server_base.cc:1034] running on GCE node
I20260808 18:28:30.540236 30136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:30.540879 30136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:30.542042 30136 hybrid_clock.cc:648] HybridClock initialized: now 1786213710542009 us; error 51 us; skew 500 ppm
Aug 08 18:28:30 dist-test-slave-gp6n krb5kdc[30018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213710, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:30.544889 30136 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:28:30.545727 30136 webserver.cc:460] Webserver started at http://127.29.78.129:39277/ using document root <none> and password file <none>
I20260808 18:28:30.546253 30136 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:30.546355 30136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:30.546607 30136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:30.548620 30136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "d77519d3cdb14abeabab20973ba0d658"
format_stamp: "Formatted at 2026-08-08 18:28:30 on dist-test-slave-gp6n"
server_key: "2200f39af402a714cb13fc091664730d"
server_key_iv: "1310cb0081ff766fba644f5ddb9e84c1"
server_key_version: "encryptionkey@0"
I20260808 18:28:30.549147 30136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "d77519d3cdb14abeabab20973ba0d658"
format_stamp: "Formatted at 2026-08-08 18:28:30 on dist-test-slave-gp6n"
server_key: "2200f39af402a714cb13fc091664730d"
server_key_iv: "1310cb0081ff766fba644f5ddb9e84c1"
server_key_version: "encryptionkey@0"
I20260808 18:28:30.552946 30136 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 18:28:30.555377 30151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:30.556582 30136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:28:30.556674 30136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "d77519d3cdb14abeabab20973ba0d658"
format_stamp: "Formatted at 2026-08-08 18:28:30 on dist-test-slave-gp6n"
server_key: "2200f39af402a714cb13fc091664730d"
server_key_iv: "1310cb0081ff766fba644f5ddb9e84c1"
server_key_version: "encryptionkey@0"
I20260808 18:28:30.556756 30136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:30.580528 30136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:30.583673 30136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:30.583834 30136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:30.584332 30136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:30.585105 30136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:30.585150 30136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:30.585188 30136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:30.585203 30136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:30.594277 30136 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:37659
I20260808 18:28:30.594319 30277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:37659 every 8 connection(s)
I20260808 18:28:30.595149 30136 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:28:30.603184 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 30136
I20260808 18:28:30.603271 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:28:30.603636 30010 external_mini_cluster.cc:1428] Setting key 082ad9b0de288d3ee139d6233c4e5927
Aug 08 18:28:30 dist-test-slave-gp6n krb5kdc[30018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213710, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:30.610018 30111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:30.596940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:34377 (local address 127.29.78.190:37375)
0808 18:28:30.597081 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:30.597085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:30.597864 (+ 779us) server_negotiation.cc:407] Connection header received
0808 18:28:30.599432 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:30.599436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:30.599496 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:30.599604 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:30.600765 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:30.601318 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:30.602504 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:30.602770 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:30.602893 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:30.605558 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:30.605562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:30.605565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:30.605592 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:30.607420 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:30.608113 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:30.608117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:30.608118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:30.608207 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:30.608706 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:30.608709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:30.608710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:30.608998 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:30.609112 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:30.609633 (+ 521us) server_negotiation.cc:299] Negotiation successful
0808 18:28:30.609779 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 18:28:30.610713 30280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:30.597268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:37375 (local address 127.29.78.129:34377)
0808 18:28:30.597705 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:30.597746 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:30.599200 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:30.599774 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:30.599781 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:30.600122 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:30.600615 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:30.600629 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:30.601459 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:30.601463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:30.602351 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:30.602359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:30.602932 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:30.602936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:30.603104 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:30.603743 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:30.603769 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:30.605389 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:30.607564 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:30.607574 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:30.607589 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:30.607980 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:30.608311 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:30.608316 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:30.608319 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:30.608521 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:30.609115 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:30.609120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:30.609382 (+ 262us) client_negotiation.cc:769] Sending connection context
0808 18:28:30.609543 (+ 161us) client_negotiation.cc:240] Negotiation successful
0808 18:28:30.609781 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":122,"threads_started":1}
I20260808 18:28:30.612054 30278 heartbeater.cc:347] Connected to a master server at 127.29.78.190:37375
I20260808 18:28:30.612346 30278 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:30.612924 30278 heartbeater.cc:511] Master 127.29.78.190:37375 requested a full tablet report, sending...
I20260808 18:28:30.615077 30068 ts_manager.cc:194] Registered new tserver with Master: d77519d3cdb14abeabab20973ba0d658 (127.29.78.129:37659)
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260808 18:28:30.616556 30068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:34377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260808 18:28:30.634617 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:37375
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:30.745308 30283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:30.745733 30283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:30.745781 30283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:30.749759 30283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:30.749815 30283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:30.749872 30283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:28:30.754875 30283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:37375
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:30.755796 30283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:30.756728 30283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:30.766685 30289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:30.766772 30283 server_base.cc:1034] running on GCE node
W20260808 18:28:30.766733 30288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:30.766942 30294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:30.767323 30283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:30.767942 30283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:30.769129 30283 hybrid_clock.cc:648] HybridClock initialized: now 1786213710769115 us; error 30 us; skew 500 ppm
Aug 08 18:28:30 dist-test-slave-gp6n krb5kdc[30018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213710, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:30.772177 30283 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:28:30.772979 30283 webserver.cc:460] Webserver started at http://127.29.78.130:43647/ using document root <none> and password file <none>
I20260808 18:28:30.773499 30283 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:30.773557 30283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:30.773716 30283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:30.775707 30283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "1f412a891b7640689aadf01aa3614a41"
format_stamp: "Formatted at 2026-08-08 18:28:30 on dist-test-slave-gp6n"
server_key: "d126aa84e6df4dd8783f14f20a0765e6"
server_key_iv: "302bf54f1374ae2b7400316842f1d346"
server_key_version: "encryptionkey@0"
I20260808 18:28:30.776204 30283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "1f412a891b7640689aadf01aa3614a41"
format_stamp: "Formatted at 2026-08-08 18:28:30 on dist-test-slave-gp6n"
server_key: "d126aa84e6df4dd8783f14f20a0765e6"
server_key_iv: "302bf54f1374ae2b7400316842f1d346"
server_key_version: "encryptionkey@0"
I20260808 18:28:30.779861 30283 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:28:30.782157 30301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:30.783404 30283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:28:30.783494 30283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "1f412a891b7640689aadf01aa3614a41"
format_stamp: "Formatted at 2026-08-08 18:28:30 on dist-test-slave-gp6n"
server_key: "d126aa84e6df4dd8783f14f20a0765e6"
server_key_iv: "302bf54f1374ae2b7400316842f1d346"
server_key_version: "encryptionkey@0"
I20260808 18:28:30.783576 30283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:30.804068 30283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:30.806962 30283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:30.807106 30283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:30.807628 30283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:30.808375 30283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:30.808436 30283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:30.808476 30283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:30.808491 30283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:30.817909 30283 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:42221
I20260808 18:28:30.817940 30422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:42221 every 8 connection(s)
I20260808 18:28:30.818889 30283 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:28:30.821159 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 30283
I20260808 18:28:30.821275 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:28:30.821609 30010 external_mini_cluster.cc:1428] Setting key fb0c80aeccf567f252153ed8202d4fcc
Aug 08 18:28:30 dist-test-slave-gp6n krb5kdc[30018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213710, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260808 18:28:30.834566 30111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:30.820927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:37193 (local address 127.29.78.190:37375)
0808 18:28:30.822119 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:30.822124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:30.822146 (+ 22us) server_negotiation.cc:407] Connection header received
0808 18:28:30.823653 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:30.823659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:30.823723 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:30.823816 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:30.824929 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:30.825653 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:30.827402 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:30.827697 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:30.827783 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:30.830793 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:30.830799 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:30.830804 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:30.830838 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:30.832544 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:30.833110 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:30.833117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:30.833122 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:30.833212 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:30.833529 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:30.833531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:30.833532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:30.833697 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:30.833774 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:30.834296 (+ 522us) server_negotiation.cc:299] Negotiation successful
0808 18:28:30.834395 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1100}
I20260808 18:28:30.835479 30426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:30.821189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:37375 (local address 127.29.78.130:37193)
0808 18:28:30.821794 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:30.821838 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:30.823467 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:30.823952 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:30.823958 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:30.824295 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:30.824780 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:30.824793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:30.825782 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:30.825789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:30.827262 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:30.827276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:30.827864 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:30.827872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:30.828120 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:30.828805 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:30.828830 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:30.830628 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:30.832666 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:30.832670 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:30.832682 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:30.832977 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:30.833308 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:30.833310 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:30.833312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:30.833443 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:30.833829 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:30.833834 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:30.834087 (+ 253us) client_negotiation.cc:769] Sending connection context
0808 18:28:30.834235 (+ 148us) client_negotiation.cc:240] Negotiation successful
0808 18:28:30.834581 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":154,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
I20260808 18:28:30.836793 30423 heartbeater.cc:347] Connected to a master server at 127.29.78.190:37375
I20260808 18:28:30.837041 30423 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:30.837549 30423 heartbeater.cc:511] Master 127.29.78.190:37375 requested a full tablet report, sending...
I20260808 18:28:30.838586 30068 ts_manager.cc:194] Registered new tserver with Master: 1f412a891b7640689aadf01aa3614a41 (127.29.78.130:42221)
I20260808 18:28:30.839468 30068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:37193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260808 18:28:30.853179 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:37375
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:30.977604 30429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:30.977873 30429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:30.977916 30429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:30.981832 30429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:30.981889 30429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:30.981977 30429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:28:30.987111 30429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:37375
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:30.988183 30429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:30.989189 30429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:30.998816 30435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:30.998862 30438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:30.998826 30434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:30.999353 30429 server_base.cc:1034] running on GCE node
I20260808 18:28:30.999675 30429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:31.000288 30429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:31.001449 30429 hybrid_clock.cc:648] HybridClock initialized: now 1786213711001429 us; error 34 us; skew 500 ppm
Aug 08 18:28:30 dist-test-slave-gp6n krb5kdc[30018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213710, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:31.004304 30429 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:28:31.005191 30429 webserver.cc:460] Webserver started at http://127.29.78.131:37345/ using document root <none> and password file <none>
I20260808 18:28:31.005735 30429 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:31.005796 30429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:31.006001 30429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:31.008112 30429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "0d800a8269104043b31d8dfe1b7474be"
format_stamp: "Formatted at 2026-08-08 18:28:30 on dist-test-slave-gp6n"
server_key: "7201fae94b00b25487642fc858916026"
server_key_iv: "63afd25630cb9184f327f08fbbded0b0"
server_key_version: "encryptionkey@0"
I20260808 18:28:31.008662 30429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "0d800a8269104043b31d8dfe1b7474be"
format_stamp: "Formatted at 2026-08-08 18:28:30 on dist-test-slave-gp6n"
server_key: "7201fae94b00b25487642fc858916026"
server_key_iv: "63afd25630cb9184f327f08fbbded0b0"
server_key_version: "encryptionkey@0"
I20260808 18:28:31.012554 30429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260808 18:28:31.014971 30446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:31.016163 30429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:28:31.016301 30429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "0d800a8269104043b31d8dfe1b7474be"
format_stamp: "Formatted at 2026-08-08 18:28:30 on dist-test-slave-gp6n"
server_key: "7201fae94b00b25487642fc858916026"
server_key_iv: "63afd25630cb9184f327f08fbbded0b0"
server_key_version: "encryptionkey@0"
I20260808 18:28:31.016410 30429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:31.034744 30429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:31.038015 30429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:31.038239 30429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:31.038899 30429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:31.039769 30429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:31.039851 30429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:31.039923 30429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:31.039965 30429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:31.050132 30429 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:36725
I20260808 18:28:31.050166 30567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:36725 every 8 connection(s)
I20260808 18:28:31.051213 30429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:28:31.060312 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 30429
I20260808 18:28:31.060431 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:28:31.060745 30010 external_mini_cluster.cc:1428] Setting key 582bd0c3612a987ead4e05e272bb4a0c
Aug 08 18:28:31 dist-test-slave-gp6n krb5kdc[30018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213710, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:31.065626 30111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:31.053034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:56055 (local address 127.29.78.190:37375)
0808 18:28:31.053183 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:31.053188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:31.054036 (+ 848us) server_negotiation.cc:407] Connection header received
0808 18:28:31.055495 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:31.055498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:31.055556 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:31.055659 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:31.056798 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:31.057369 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:31.058572 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:31.058798 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:31.058909 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:31.061652 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:31.061656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:31.061659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:31.061685 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:31.063587 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:31.064081 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:31.064086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:31.064088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:31.064179 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:31.064536 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:31.064539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:31.064541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:31.064730 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:31.064834 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:31.065327 (+ 493us) server_negotiation.cc:299] Negotiation successful
0808 18:28:31.065447 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 18:28:31.066520 30570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:31.053401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:37375 (local address 127.29.78.131:56055)
0808 18:28:31.053865 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:31.053928 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:31.055278 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:31.055828 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:31.055835 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:31.056189 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:31.056658 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:31.056670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:31.057503 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:31.057507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:31.058381 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:31.058390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:31.058945 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:31.058950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:31.059120 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:31.059698 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:31.059713 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:31.061492 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:31.063689 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:31.063694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:31.063709 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:31.063967 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:31.064279 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:31.064282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:31.064284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:31.064425 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:31.064847 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:31.064855 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:31.065099 (+ 244us) client_negotiation.cc:769] Sending connection context
0808 18:28:31.065284 (+ 185us) client_negotiation.cc:240] Negotiation successful
0808 18:28:31.065594 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":106,"threads_started":1}
I20260808 18:28:31.067934 30568 heartbeater.cc:347] Connected to a master server at 127.29.78.190:37375
I20260808 18:28:31.068146 30568 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:31.068599 30568 heartbeater.cc:511] Master 127.29.78.190:37375 requested a full tablet report, sending...
I20260808 18:28:31.069603 30068 ts_manager.cc:194] Registered new tserver with Master: 0d800a8269104043b31d8dfe1b7474be (127.29.78.131:36725)
I20260808 18:28:31.070218 30068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:56055
I20260808 18:28:31.075943 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 18:28:31 dist-test-slave-gp6n krb5kdc[30018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 18:28:31 dist-test-slave-gp6n krb5kdc[30018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260808 18:28:31.104498 30576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:31.094620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58968 (local address 127.29.78.129:37659)
0808 18:28:31.094949 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:31.094963 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:31.094987 (+ 24us) server_negotiation.cc:407] Connection header received
0808 18:28:31.095039 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:31.095044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:31.095241 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:31.095409 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:31.096070 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:31.096961 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:31.097773 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:31.098177 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:31.098266 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:31.100819 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:31.100828 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:31.100842 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:31.100866 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:31.102770 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:31.103185 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:31.103197 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:31.103198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:31.103291 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:31.103528 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:31.103536 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:31.103537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:31.103774 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:31.103956 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:31.104085 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 18:28:31.104258 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":124,"threads_started":1}
W20260808 18:28:31.105490 30181 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58968
Aug 08 18:28:31 dist-test-slave-gp6n krb5kdc[30018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 18:28:31 dist-test-slave-gp6n krb5kdc[30018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260808 18:28:31.133510 30576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:31.124584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.29.78.129:37659)
0808 18:28:31.124707 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:31.124711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:31.124821 (+ 110us) server_negotiation.cc:407] Connection header received
0808 18:28:31.124909 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:31.124911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:31.124963 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:31.125051 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:31.125791 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:31.126366 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:31.127424 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:31.127707 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:31.127798 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:31.130333 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:31.130336 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:31.130338 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:31.130464 (+ 126us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:31.131986 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:31.132352 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:31.132354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:31.132355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:31.132469 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:31.132691 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:31.132693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:31.132695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:31.132849 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:31.132946 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:31.133156 (+ 210us) server_negotiation.cc:299] Negotiation successful
0808 18:28:31.133254 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 18:28:31.134603 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 30136
I20260808 18:28:31.142292 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 30283
I20260808 18:28:31.149945 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 30429
I20260808 18:28:31.156992 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 30032
2026-08-08T18:28:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2375 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:28:31 dist-test-slave-gp6n krb5kdc[30583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:28:31 dist-test-slave-gp6n krb5kdc[30583](info): set up 2 sockets
Aug 08 18:28:31 dist-test-slave-gp6n krb5kdc[30583](info): commencing operation
krb5kdc: starting...
W20260808 18:28:32.250029 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:28:32 dist-test-slave-gp6n krb5kdc[30583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:28:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:28:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:28:32.377825 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:34853
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:34853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:32.488739 30598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:32.489037 30598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:32.489116 30598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:32.493162 30598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:28:32.493249 30598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:32.493288 30598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:32.493350 30598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:28:32.493388 30598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:28:32.498652 30598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:34853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:34853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:32.499681 30598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:32.500705 30598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:32.509261 30604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:32.509289 30603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:32.509371 30598 server_base.cc:1034] running on GCE node
W20260808 18:28:32.509260 30606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:32.509977 30598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:32.511155 30598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:32.512358 30598 hybrid_clock.cc:648] HybridClock initialized: now 1786213712512338 us; error 31 us; skew 500 ppm
Aug 08 18:28:32 dist-test-slave-gp6n krb5kdc[30583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213712, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:32.515103 30598 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:28:32.515888 30598 webserver.cc:460] Webserver started at http://127.29.78.190:35153/ using document root <none> and password file <none>
I20260808 18:28:32.516382 30598 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:32.516439 30598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:32.516606 30598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:32.518576 30598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "e7a5c78b482f4aeab0cc12c915f3d917"
format_stamp: "Formatted at 2026-08-08 18:28:32 on dist-test-slave-gp6n"
server_key: "78c24501cb7d094d26539c237b163084"
server_key_iv: "574d24733451dac9e8002b39a2c7fc93"
server_key_version: "encryptionkey@0"
I20260808 18:28:32.519088 30598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "e7a5c78b482f4aeab0cc12c915f3d917"
format_stamp: "Formatted at 2026-08-08 18:28:32 on dist-test-slave-gp6n"
server_key: "78c24501cb7d094d26539c237b163084"
server_key_iv: "574d24733451dac9e8002b39a2c7fc93"
server_key_version: "encryptionkey@0"
I20260808 18:28:32.522598 30598 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 18:28:32.524807 30613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:32.525971 30598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:28:32.526090 30598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "e7a5c78b482f4aeab0cc12c915f3d917"
format_stamp: "Formatted at 2026-08-08 18:28:32 on dist-test-slave-gp6n"
server_key: "78c24501cb7d094d26539c237b163084"
server_key_iv: "574d24733451dac9e8002b39a2c7fc93"
server_key_version: "encryptionkey@0"
I20260808 18:28:32.526191 30598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:32.539719 30598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:32.542794 30598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:32.543006 30598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:32.550690 30598 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:34853
I20260808 18:28:32.550734 30671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:34853 every 8 connection(s)
I20260808 18:28:32.551718 30598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:28:32.553793 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 30598
I20260808 18:28:32.553892 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:28:32.554199 30010 external_mini_cluster.cc:1428] Setting key 52e86f2be15723670c79b609513c1aae
I20260808 18:28:32.554605 30673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:32.561542 30673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917: Bootstrap starting.
I20260808 18:28:32.563687 30673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917: Neither blocks nor log segments found. Creating new log.
Aug 08 18:28:32 dist-test-slave-gp6n krb5kdc[30583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:32.564604 30673 log.cc:824] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:32.566896 30673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917: No bootstrap required, opened a new log
I20260808 18:28:32.569957 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:32.555599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51220 (local address 127.29.78.190:34853)
0808 18:28:32.556057 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:32.556078 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:32.556120 (+ 42us) server_negotiation.cc:407] Connection header received
0808 18:28:32.556890 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:32.556912 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:32.557317 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:32.557672 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:32.558439 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:32.559907 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:32.560829 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:32.561493 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:32.561654 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:32.564562 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:32.564568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:32.564581 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:32.564609 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:32.566710 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:32.567212 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:32.567220 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:32.567225 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:32.567366 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:32.567634 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:32.567638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:32.567639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:32.568123 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:32.568316 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:32.568570 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 18:28:32.568871 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":138,"threads_started":1}
I20260808 18:28:32.571590 30673 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5c78b482f4aeab0cc12c915f3d917" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34853 } }
I20260808 18:28:32.571789 30673 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:32.571848 30673 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a5c78b482f4aeab0cc12c915f3d917, State: Initialized, Role: FOLLOWER
I20260808 18:28:32.572309 30673 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5c78b482f4aeab0cc12c915f3d917" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34853 } }
I20260808 18:28:32.572458 30673 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:28:32.572499 30673 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:28:32.572620 30673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:32.573450 30673 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5c78b482f4aeab0cc12c915f3d917" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34853 } }
I20260808 18:28:32.573781 30673 leader_election.cc:302] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7a5c78b482f4aeab0cc12c915f3d917; no voters:
I20260808 18:28:32.574065 30673 leader_election.cc:288] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:28:32.574194 30678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:32.574472 30678 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917 [term 1 LEADER]: Becoming Leader. State: Replica: e7a5c78b482f4aeab0cc12c915f3d917, State: Running, Role: LEADER
I20260808 18:28:32.574803 30678 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5c78b482f4aeab0cc12c915f3d917" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34853 } }
I20260808 18:28:32.575254 30673 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:28:32.576350 30680 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7a5c78b482f4aeab0cc12c915f3d917. Latest consensus state: current_term: 1 leader_uuid: "e7a5c78b482f4aeab0cc12c915f3d917" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5c78b482f4aeab0cc12c915f3d917" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34853 } } }
I20260808 18:28:32.576494 30680 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917 [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:32.576812 30679 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7a5c78b482f4aeab0cc12c915f3d917" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5c78b482f4aeab0cc12c915f3d917" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34853 } } }
I20260808 18:28:32.576911 30689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:28:32.576922 30679 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917 [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:32.579932 30689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:28:32.585646 30689 catalog_manager.cc:1347] Generated new cluster ID: 6cd5f16483b74708ad91dab8c0f8d51f
I20260808 18:28:32.585724 30689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:28:32.613480 30689 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:28:32.614537 30689 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:28:32.620461 30689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7a5c78b482f4aeab0cc12c915f3d917: Generated new TSK 0
I20260808 18:28:32.621169 30689 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260808 18:28:32.651881 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34853
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:32.761245 30701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:32.761492 30701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:32.761543 30701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:32.765477 30701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:32.765537 30701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:32.765630 30701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:28:32.770687 30701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34853
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:32.771646 30701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:32.772648 30701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:32.782354 30711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:32.782363 30709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:32.782603 30701 server_base.cc:1034] running on GCE node
W20260808 18:28:32.782347 30707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:32.783109 30701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:32.783702 30701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:32.784883 30701 hybrid_clock.cc:648] HybridClock initialized: now 1786213712784851 us; error 43 us; skew 500 ppm
Aug 08 18:28:32 dist-test-slave-gp6n krb5kdc[30583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213712, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:32.787616 30701 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:28:32.788431 30701 webserver.cc:460] Webserver started at http://127.29.78.129:36371/ using document root <none> and password file <none>
I20260808 18:28:32.788969 30701 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:32.789067 30701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:32.789288 30701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:32.791349 30701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "25601140ec2a431380c5523a3ab49830"
format_stamp: "Formatted at 2026-08-08 18:28:32 on dist-test-slave-gp6n"
server_key: "5723116d28e290d6caa5931f7e194060"
server_key_iv: "fe83203666d5d974a6acf82fe92780f0"
server_key_version: "encryptionkey@0"
I20260808 18:28:32.791909 30701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "25601140ec2a431380c5523a3ab49830"
format_stamp: "Formatted at 2026-08-08 18:28:32 on dist-test-slave-gp6n"
server_key: "5723116d28e290d6caa5931f7e194060"
server_key_iv: "fe83203666d5d974a6acf82fe92780f0"
server_key_version: "encryptionkey@0"
I20260808 18:28:32.795576 30701 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260808 18:28:32.797873 30718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:32.799048 30701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:28:32.799180 30701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "25601140ec2a431380c5523a3ab49830"
format_stamp: "Formatted at 2026-08-08 18:28:32 on dist-test-slave-gp6n"
server_key: "5723116d28e290d6caa5931f7e194060"
server_key_iv: "fe83203666d5d974a6acf82fe92780f0"
server_key_version: "encryptionkey@0"
I20260808 18:28:32.799288 30701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:32.813966 30701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:32.817034 30701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:32.817250 30701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:32.817854 30701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:32.818763 30701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:32.818835 30701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:32.818907 30701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:32.818953 30701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:32.828106 30701 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:36613
I20260808 18:28:32.828126 30845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:36613 every 8 connection(s)
I20260808 18:28:32.829039 30701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
Aug 08 18:28:32 dist-test-slave-gp6n krb5kdc[30583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213712, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:32.838475 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 30701
I20260808 18:28:32.838586 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:28:32.838842 30010 external_mini_cluster.cc:1428] Setting key 7d093b4702c8bafce08fb93554336a4a
I20260808 18:28:32.842800 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:32.830893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:54783 (local address 127.29.78.190:34853)
0808 18:28:32.831044 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:32.831048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:32.831836 (+ 788us) server_negotiation.cc:407] Connection header received
0808 18:28:32.833106 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:32.833109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:32.833156 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:32.833231 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:32.834352 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:32.834944 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:32.836045 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:32.836265 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:32.836340 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:32.839027 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:32.839031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:32.839033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:32.839057 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:32.840712 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:32.841232 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:32.841236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:32.841239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:32.841326 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:32.841670 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:32.841673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:32.841675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:32.841865 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:32.841973 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:32.842489 (+ 516us) server_negotiation.cc:299] Negotiation successful
0808 18:28:32.842630 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 18:28:32.843712 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:32.831249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34853 (local address 127.29.78.129:54783)
0808 18:28:32.831676 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:32.831718 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:32.832899 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:32.833358 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:32.833365 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:32.833717 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:32.834208 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:32.834228 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:32.835052 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:32.835055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:32.835929 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:32.835937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:32.836391 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:32.836393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:32.836531 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:32.837149 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:32.837166 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:32.838842 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:32.840831 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:32.840837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:32.840853 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:32.841126 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:32.841417 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:32.841421 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:32.841423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:32.841572 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:32.841971 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:32.841978 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:32.842234 (+ 256us) client_negotiation.cc:769] Sending connection context
0808 18:28:32.842454 (+ 220us) client_negotiation.cc:240] Negotiation successful
0808 18:28:32.842786 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":110,"threads_started":1}
I20260808 18:28:32.844987 30846 heartbeater.cc:347] Connected to a master server at 127.29.78.190:34853
I20260808 18:28:32.845264 30846 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:32.845906 30846 heartbeater.cc:511] Master 127.29.78.190:34853 requested a full tablet report, sending...
I20260808 18:28:32.847594 30632 ts_manager.cc:194] Registered new tserver with Master: 25601140ec2a431380c5523a3ab49830 (127.29.78.129:36613)
I20260808 18:28:32.848732 30632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:54783
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260808 18:28:32.869864 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34853
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:32.980597 30852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:32.980873 30852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:32.980953 30852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:32.985077 30852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:32.985162 30852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:32.985255 30852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:28:32.990501 30852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34853
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:32.991506 30852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:32.992507 30852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:33.002173 30860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:33.002211 30858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:33.002238 30857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:33.002442 30852 server_base.cc:1034] running on GCE node
I20260808 18:28:33.003082 30852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:33.003749 30852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:33.004914 30852 hybrid_clock.cc:648] HybridClock initialized: now 1786213713004898 us; error 26 us; skew 500 ppm
Aug 08 18:28:33 dist-test-slave-gp6n krb5kdc[30583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213713, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:33.007714 30852 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:28:33.008543 30852 webserver.cc:460] Webserver started at http://127.29.78.130:40495/ using document root <none> and password file <none>
I20260808 18:28:33.009040 30852 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:33.009096 30852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:33.009260 30852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:33.011332 30852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "8a4237a501ca4074a01bf24b05d55c09"
format_stamp: "Formatted at 2026-08-08 18:28:33 on dist-test-slave-gp6n"
server_key: "36812b516530459e47116f3c805bb712"
server_key_iv: "6149a4cd116f774c66d0a5c40d2854f8"
server_key_version: "encryptionkey@0"
I20260808 18:28:33.011876 30852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "8a4237a501ca4074a01bf24b05d55c09"
format_stamp: "Formatted at 2026-08-08 18:28:33 on dist-test-slave-gp6n"
server_key: "36812b516530459e47116f3c805bb712"
server_key_iv: "6149a4cd116f774c66d0a5c40d2854f8"
server_key_version: "encryptionkey@0"
I20260808 18:28:33.015440 30852 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:28:33.017726 30869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:33.018942 30852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:28:33.019062 30852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "8a4237a501ca4074a01bf24b05d55c09"
format_stamp: "Formatted at 2026-08-08 18:28:33 on dist-test-slave-gp6n"
server_key: "36812b516530459e47116f3c805bb712"
server_key_iv: "6149a4cd116f774c66d0a5c40d2854f8"
server_key_version: "encryptionkey@0"
I20260808 18:28:33.019161 30852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:33.031800 30852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:33.035121 30852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:33.035320 30852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:33.035952 30852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:33.036784 30852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:33.036837 30852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:33.036922 30852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:33.036960 30852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:33.045581 30852 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:33611
I20260808 18:28:33.045603 30991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:33611 every 8 connection(s)
I20260808 18:28:33.046908 30852 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:28:33.056308 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 30852
I20260808 18:28:33.056406 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:28:33.056677 30010 external_mini_cluster.cc:1428] Setting key 1cab017b4f1a6fb46d3b4516aa719d38
Aug 08 18:28:33 dist-test-slave-gp6n krb5kdc[30583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213713, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:33.061802 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:33.048843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:57927 (local address 127.29.78.190:34853)
0808 18:28:33.048988 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:33.048992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:33.049994 (+ 1002us) server_negotiation.cc:407] Connection header received
0808 18:28:33.051569 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:33.051572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:33.051619 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:33.051701 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:33.052977 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.053517 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.054963 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.055189 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.055269 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:33.057926 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:33.057931 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:33.057935 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:33.057965 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:33.059516 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:33.060144 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:33.060149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:33.060152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:33.060255 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:33.060592 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:33.060597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:33.060599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:33.060855 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:33.060971 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:33.061426 (+ 455us) server_negotiation.cc:299] Negotiation successful
0808 18:28:33.061568 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 18:28:33.062559 30995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:33.049201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34853 (local address 127.29.78.130:57927)
0808 18:28:33.049808 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:33.049859 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:33.051267 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:33.051910 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:33.051917 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:33.052298 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:33.052798 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:33.052812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.053813 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.053816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:33.054829 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:33.054839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.055320 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.055322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:33.055495 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:33.056025 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:33.056041 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:33.057756 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:33.059632 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:33.059640 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:33.059658 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:33.060028 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:33.060359 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:33.060362 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:33.060363 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:33.060495 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:33.060966 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:33.060971 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:33.061191 (+ 220us) client_negotiation.cc:769] Sending connection context
0808 18:28:33.061342 (+ 151us) client_negotiation.cc:240] Negotiation successful
0808 18:28:33.061611 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":139,"threads_started":1}
I20260808 18:28:33.063694 30992 heartbeater.cc:347] Connected to a master server at 127.29.78.190:34853
I20260808 18:28:33.063939 30992 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:33.064419 30992 heartbeater.cc:511] Master 127.29.78.190:34853 requested a full tablet report, sending...
I20260808 18:28:33.065485 30632 ts_manager.cc:194] Registered new tserver with Master: 8a4237a501ca4074a01bf24b05d55c09 (127.29.78.130:33611)
I20260808 18:28:33.066311 30632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:57927
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260808 18:28:33.086618 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34853
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:33.197024 30999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:33.197317 30999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:33.197394 30999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:33.201426 30999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:33.201514 30999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:33.201607 30999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:28:33.206835 30999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34853
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:33.207858 30999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:33.208870 30999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:28:33.218564 30999 server_base.cc:1034] running on GCE node
W20260808 18:28:33.218506 31005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:33.218506 31007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:33.218540 31004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:33.219126 30999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:33.219775 30999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:33.220958 30999 hybrid_clock.cc:648] HybridClock initialized: now 1786213713220946 us; error 34 us; skew 500 ppm
Aug 08 18:28:33 dist-test-slave-gp6n krb5kdc[30583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213713, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:33.223806 30999 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:28:33.224596 30999 webserver.cc:460] Webserver started at http://127.29.78.131:33693/ using document root <none> and password file <none>
I20260808 18:28:33.225106 30999 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:33.225162 30999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:33.225322 30999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:33.227311 30999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "375798fdad754e199f171906607f831e"
format_stamp: "Formatted at 2026-08-08 18:28:33 on dist-test-slave-gp6n"
server_key: "3a0a290eb504636ce0bd64d372a6cfb2"
server_key_iv: "d04904054b291bed4712e3735d7db6a8"
server_key_version: "encryptionkey@0"
I20260808 18:28:33.227820 30999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "375798fdad754e199f171906607f831e"
format_stamp: "Formatted at 2026-08-08 18:28:33 on dist-test-slave-gp6n"
server_key: "3a0a290eb504636ce0bd64d372a6cfb2"
server_key_iv: "d04904054b291bed4712e3735d7db6a8"
server_key_version: "encryptionkey@0"
I20260808 18:28:33.231451 30999 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260808 18:28:33.233675 31014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:33.234884 30999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260808 18:28:33.235008 30999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "375798fdad754e199f171906607f831e"
format_stamp: "Formatted at 2026-08-08 18:28:33 on dist-test-slave-gp6n"
server_key: "3a0a290eb504636ce0bd64d372a6cfb2"
server_key_iv: "d04904054b291bed4712e3735d7db6a8"
server_key_version: "encryptionkey@0"
I20260808 18:28:33.235131 30999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:33.282294 30999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:33.285403 30999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:33.285629 30999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:33.286262 30999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:33.287127 30999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:33.287206 30999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:33.287282 30999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:33.287338 30999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:33.296051 30999 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:33667
I20260808 18:28:33.296080 31143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:33667 every 8 connection(s)
I20260808 18:28:33.296929 30999 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:28:33.304064 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 30999
I20260808 18:28:33.304206 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:28:33.304526 30010 external_mini_cluster.cc:1428] Setting key 102003249f2e4946ca974ef9588ce598
Aug 08 18:28:33 dist-test-slave-gp6n krb5kdc[30583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213713, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:33.310636 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:33.298818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:53157 (local address 127.29.78.190:34853)
0808 18:28:33.298959 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:33.298963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:33.299767 (+ 804us) server_negotiation.cc:407] Connection header received
0808 18:28:33.301089 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:33.301092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:33.301153 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:33.301263 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:33.302398 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.303052 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.304296 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.304639 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.304752 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:33.307643 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:33.307646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:33.307648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:33.307670 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:33.308863 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:33.309317 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:33.309320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:33.309321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:33.309392 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:33.309692 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:33.309694 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:33.309695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:33.309852 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:33.309927 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:33.310351 (+ 424us) server_negotiation.cc:299] Negotiation successful
0808 18:28:33.310521 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 18:28:33.311376 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:33.299191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34853 (local address 127.29.78.131:53157)
0808 18:28:33.299621 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:33.299662 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:33.300872 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:33.301433 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:33.301439 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:33.301782 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:33.302250 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:33.302264 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.303182 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.303192 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:33.304131 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:33.304140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.304780 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.304783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:33.305004 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:33.305740 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:33.305761 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:33.307494 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:33.308961 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:33.308965 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:33.308985 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:33.309222 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:33.309487 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:33.309489 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:33.309491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:33.309613 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:33.309937 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:33.309941 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:33.310157 (+ 216us) client_negotiation.cc:769] Sending connection context
0808 18:28:33.310303 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 18:28:33.310592 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":121,"threads_started":1}
I20260808 18:28:33.312584 31144 heartbeater.cc:347] Connected to a master server at 127.29.78.190:34853
I20260808 18:28:33.312810 31144 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:33.313268 31144 heartbeater.cc:511] Master 127.29.78.190:34853 requested a full tablet report, sending...
I20260808 18:28:33.314293 30632 ts_manager.cc:194] Registered new tserver with Master: 375798fdad754e199f171906607f831e (127.29.78.131:33667)
I20260808 18:28:33.314852 30632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:53157
I20260808 18:28:33.318341 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:28:33.329160 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:33.322460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51226 (local address 127.29.78.190:34853)
0808 18:28:33.322582 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:33.322586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:33.322746 (+ 160us) server_negotiation.cc:407] Connection header received
0808 18:28:33.322819 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:33.322821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:33.322856 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:33.322935 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:33.323683 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.324177 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.324936 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.325133 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.325205 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:33.326595 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:33.326598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:33.326600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:33.326618 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:33.327809 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:33.328183 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:33.328185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:33.328186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:33.328257 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:33.328481 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:33.328483 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:33.328484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:33.328668 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:33.328769 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:33.328968 (+ 199us) server_negotiation.cc:299] Negotiation successful
0808 18:28:33.329055 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 18:28:33.334486 30632 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:28:33.337582 30632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:28:33.352993 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:33.347317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:33611 (local address 127.0.0.1:52802)
0808 18:28:33.348535 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:33.348549 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:33.348670 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:33.349155 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:33.349161 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:33.349182 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:33.349341 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:33.349349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.350436 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.350440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:33.351638 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:33.351648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.352414 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.352418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:33.352547 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:33.352587 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 18:28:33.352705 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 18:28:33.352823 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1151,"thread_start_us":96,"threads_started":1}
I20260808 18:28:33.352994 31162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:33.348052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52802 (local address 127.29.78.130:33611)
0808 18:28:33.348390 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:33.348403 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:33.348646 (+ 243us) server_negotiation.cc:407] Connection header received
0808 18:28:33.348823 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:33.348832 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:33.349002 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:33.349167 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:33.349464 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.350285 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.351757 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.352289 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.352414 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:33.352631 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:33.352732 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 18:28:33.352823 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":119,"threads_started":1}
I20260808 18:28:33.353937 31160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:33.347241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:33667 (local address 127.0.0.1:33088)
0808 18:28:33.348094 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:33.348145 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:33.348271 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:33.349499 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:33.349502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:33.349524 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:33.349600 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:33.349606 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.351017 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.351020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:33.352481 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:33.352493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.353537 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.353541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:33.353635 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:33.353660 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 18:28:33.353746 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 18:28:33.353814 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":191,"thread_start_us":86,"threads_started":1}
I20260808 18:28:33.354027 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:33.347342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.29.78.131:33667)
0808 18:28:33.348924 (+ 1582us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:33.348940 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:33.348959 (+ 19us) server_negotiation.cc:407] Connection header received
0808 18:28:33.349022 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:33.349026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:33.349179 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:33.349321 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:33.349701 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.350871 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.352627 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.353435 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.353539 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:33.353663 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:33.353772 (+ 109us) server_negotiation.cc:299] Negotiation successful
0808 18:28:33.353876 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1481,"thread_start_us":135,"threads_started":1}
I20260808 18:28:33.354121 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:33.347639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:36613 (local address 127.0.0.1:43866)
0808 18:28:33.348630 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:33.348642 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:33.348753 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:33.349393 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:33.349396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:33.349416 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:33.349520 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:33.349527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.351017 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.351020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:33.352487 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:33.352493 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.353732 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.353735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:33.353822 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:33.353845 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 18:28:33.353915 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 18:28:33.353977 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"mutex_wait_us":155,"thread_start_us":106,"threads_started":1}
I20260808 18:28:33.354537 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:33.347859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43866 (local address 127.29.78.129:36613)
0808 18:28:33.348484 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:33.348489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:33.348726 (+ 237us) server_negotiation.cc:407] Connection header received
0808 18:28:33.348883 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:33.348890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:33.349080 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:33.349221 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:33.349649 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.350871 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.352627 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.353435 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.353539 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:33.353663 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:33.354218 (+ 555us) server_negotiation.cc:299] Negotiation successful
0808 18:28:33.354323 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":76,"threads_started":1}
I20260808 18:28:33.356566 30923 tablet_service.cc:1467] Processing CreateTablet for tablet f7e78357a87b4ee39e4628fa8b3e4ec5 (DEFAULT_TABLE table=test-table [id=bfe7aa0504b84ffe893ae3b0f3ab0c82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:33.357034 30772 tablet_service.cc:1467] Processing CreateTablet for tablet f7e78357a87b4ee39e4628fa8b3e4ec5 (DEFAULT_TABLE table=test-table [id=bfe7aa0504b84ffe893ae3b0f3ab0c82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:33.356966 31069 tablet_service.cc:1467] Processing CreateTablet for tablet f7e78357a87b4ee39e4628fa8b3e4ec5 (DEFAULT_TABLE table=test-table [id=bfe7aa0504b84ffe893ae3b0f3ab0c82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:33.357594 30923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7e78357a87b4ee39e4628fa8b3e4ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:33.357992 31069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7e78357a87b4ee39e4628fa8b3e4ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:33.357990 30772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7e78357a87b4ee39e4628fa8b3e4ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:33.364264 31163 tablet_bootstrap.cc:492] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09: Bootstrap starting.
I20260808 18:28:33.365306 31165 tablet_bootstrap.cc:492] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 375798fdad754e199f171906607f831e: Bootstrap starting.
I20260808 18:28:33.365609 31164 tablet_bootstrap.cc:492] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 25601140ec2a431380c5523a3ab49830: Bootstrap starting.
I20260808 18:28:33.366782 31163 tablet_bootstrap.cc:654] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:33.367452 31165 tablet_bootstrap.cc:654] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 375798fdad754e199f171906607f831e: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:33.367452 31164 tablet_bootstrap.cc:654] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 25601140ec2a431380c5523a3ab49830: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:33.367532 31163 log.cc:824] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:33.368408 31164 log.cc:824] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 25601140ec2a431380c5523a3ab49830: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:33.368700 31165 log.cc:824] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 375798fdad754e199f171906607f831e: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:33.369539 31163 tablet_bootstrap.cc:492] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09: No bootstrap required, opened a new log
I20260808 18:28:33.369789 31163 ts_tablet_manager.cc:1397] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260808 18:28:33.370594 31164 tablet_bootstrap.cc:492] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 25601140ec2a431380c5523a3ab49830: No bootstrap required, opened a new log
I20260808 18:28:33.370810 31164 ts_tablet_manager.cc:1397] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 25601140ec2a431380c5523a3ab49830: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260808 18:28:33.371002 31165 tablet_bootstrap.cc:492] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 375798fdad754e199f171906607f831e: No bootstrap required, opened a new log
I20260808 18:28:33.371210 31165 ts_tablet_manager.cc:1397] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 375798fdad754e199f171906607f831e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:28:33.373905 31163 raft_consensus.cc:348] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375798fdad754e199f171906607f831e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33667 } } peers { permanent_uuid: "8a4237a501ca4074a01bf24b05d55c09" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33611 } } peers { permanent_uuid: "25601140ec2a431380c5523a3ab49830" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36613 } }
I20260808 18:28:33.374197 31163 raft_consensus.cc:374] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:33.374285 31163 raft_consensus.cc:729] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4237a501ca4074a01bf24b05d55c09, State: Initialized, Role: FOLLOWER
I20260808 18:28:33.374102 31165 raft_consensus.cc:348] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 375798fdad754e199f171906607f831e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375798fdad754e199f171906607f831e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33667 } } peers { permanent_uuid: "8a4237a501ca4074a01bf24b05d55c09" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33611 } } peers { permanent_uuid: "25601140ec2a431380c5523a3ab49830" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36613 } }
I20260808 18:28:33.374449 31165 raft_consensus.cc:374] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 375798fdad754e199f171906607f831e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:33.374517 31165 raft_consensus.cc:729] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 375798fdad754e199f171906607f831e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375798fdad754e199f171906607f831e, State: Initialized, Role: FOLLOWER
I20260808 18:28:33.374769 31163 consensus_queue.cc:260] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375798fdad754e199f171906607f831e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33667 } } peers { permanent_uuid: "8a4237a501ca4074a01bf24b05d55c09" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33611 } } peers { permanent_uuid: "25601140ec2a431380c5523a3ab49830" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36613 } }
I20260808 18:28:33.375025 31165 consensus_queue.cc:260] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 375798fdad754e199f171906607f831e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375798fdad754e199f171906607f831e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33667 } } peers { permanent_uuid: "8a4237a501ca4074a01bf24b05d55c09" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33611 } } peers { permanent_uuid: "25601140ec2a431380c5523a3ab49830" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36613 } }
I20260808 18:28:33.375138 31164 raft_consensus.cc:348] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 25601140ec2a431380c5523a3ab49830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375798fdad754e199f171906607f831e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33667 } } peers { permanent_uuid: "8a4237a501ca4074a01bf24b05d55c09" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33611 } } peers { permanent_uuid: "25601140ec2a431380c5523a3ab49830" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36613 } }
I20260808 18:28:33.375447 31164 raft_consensus.cc:374] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 25601140ec2a431380c5523a3ab49830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:33.375511 30992 heartbeater.cc:503] Master 127.29.78.190:34853 was elected leader, sending a full tablet report...
I20260808 18:28:33.375530 31164 raft_consensus.cc:729] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 25601140ec2a431380c5523a3ab49830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25601140ec2a431380c5523a3ab49830, State: Initialized, Role: FOLLOWER
I20260808 18:28:33.375841 31163 ts_tablet_manager.cc:1428] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:28:33.376179 31165 ts_tablet_manager.cc:1428] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 375798fdad754e199f171906607f831e: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260808 18:28:33.376271 31144 heartbeater.cc:503] Master 127.29.78.190:34853 was elected leader, sending a full tablet report...
I20260808 18:28:33.376799 31164 consensus_queue.cc:260] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 25601140ec2a431380c5523a3ab49830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375798fdad754e199f171906607f831e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33667 } } peers { permanent_uuid: "8a4237a501ca4074a01bf24b05d55c09" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33611 } } peers { permanent_uuid: "25601140ec2a431380c5523a3ab49830" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36613 } }
I20260808 18:28:33.377446 30846 heartbeater.cc:503] Master 127.29.78.190:34853 was elected leader, sending a full tablet report...
I20260808 18:28:33.377779 31164 ts_tablet_manager.cc:1428] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 25601140ec2a431380c5523a3ab49830: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260808 18:28:33.435297 31169 raft_consensus.cc:482] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:28:33.435559 31169 raft_consensus.cc:504] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375798fdad754e199f171906607f831e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33667 } } peers { permanent_uuid: "8a4237a501ca4074a01bf24b05d55c09" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33611 } } peers { permanent_uuid: "25601140ec2a431380c5523a3ab49830" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36613 } }
I20260808 18:28:33.436707 31169 leader_election.cc:288] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 375798fdad754e199f171906607f831e (127.29.78.131:33667), 25601140ec2a431380c5523a3ab49830 (127.29.78.129:36613)
I20260808 18:28:33.442319 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:33.437074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:52849 (local address 127.29.78.131:33667)
0808 18:28:33.437190 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:33.437197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:33.437269 (+ 72us) server_negotiation.cc:407] Connection header received
0808 18:28:33.437407 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:33.437410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:33.437453 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:33.437661 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:33.438040 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.438884 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.440898 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.441563 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.441696 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:33.441748 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:33.442118 (+ 370us) server_negotiation.cc:299] Negotiation successful
0808 18:28:33.442185 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 18:28:33.442370 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:33.437110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:43131 (local address 127.29.78.129:36613)
0808 18:28:33.437239 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:33.437246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:33.437519 (+ 273us) server_negotiation.cc:407] Connection header received
0808 18:28:33.437632 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:33.437637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:33.437703 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:33.437780 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:33.438040 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.438885 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.440898 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.441570 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.441695 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:33.441748 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:33.442108 (+ 360us) server_negotiation.cc:299] Negotiation successful
0808 18:28:33.442185 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 18:28:33.442451 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:33.436985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:36613 (local address 127.29.78.130:43131)
0808 18:28:33.437398 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:33.437413 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:33.437506 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:33.437796 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:33.437799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:33.437811 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:33.437920 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:33.437929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.439038 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.439043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:33.440732 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:33.440745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.441718 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.441723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:33.441944 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:33.441992 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 18:28:33.442079 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 18:28:33.442181 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":251,"thread_start_us":121,"threads_started":1}
I20260808 18:28:33.442911 30995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:33.436962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:33667 (local address 127.29.78.130:52849)
0808 18:28:33.437164 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:33.437180 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:33.437296 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:33.437708 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:33.437712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:33.437733 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:33.437880 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:33.437889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.439038 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.439043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:33.440732 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:33.440745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.441718 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.441723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:33.441939 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:33.441986 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 18:28:33.442096 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 18:28:33.442180 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":163}
I20260808 18:28:33.443457 30795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e78357a87b4ee39e4628fa8b3e4ec5" candidate_uuid: "8a4237a501ca4074a01bf24b05d55c09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25601140ec2a431380c5523a3ab49830" is_pre_election: true
I20260808 18:28:33.443682 31089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e78357a87b4ee39e4628fa8b3e4ec5" candidate_uuid: "8a4237a501ca4074a01bf24b05d55c09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "375798fdad754e199f171906607f831e" is_pre_election: true
I20260808 18:28:33.443928 30795 raft_consensus.cc:2454] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 25601140ec2a431380c5523a3ab49830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a4237a501ca4074a01bf24b05d55c09 in term 0.
I20260808 18:28:33.444011 31089 raft_consensus.cc:2454] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 375798fdad754e199f171906607f831e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a4237a501ca4074a01bf24b05d55c09 in term 0.
I20260808 18:28:33.444568 30872 leader_election.cc:302] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 25601140ec2a431380c5523a3ab49830, 375798fdad754e199f171906607f831e, 8a4237a501ca4074a01bf24b05d55c09; no voters:
I20260808 18:28:33.444845 31169 raft_consensus.cc:2781] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:28:33.444929 31169 raft_consensus.cc:482] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:28:33.444972 31169 raft_consensus.cc:3037] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:33.446010 31169 raft_consensus.cc:504] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375798fdad754e199f171906607f831e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33667 } } peers { permanent_uuid: "8a4237a501ca4074a01bf24b05d55c09" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33611 } } peers { permanent_uuid: "25601140ec2a431380c5523a3ab49830" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36613 } }
I20260808 18:28:33.446375 31169 leader_election.cc:288] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09 [CANDIDATE]: Term 1 election: Requested vote from peers 375798fdad754e199f171906607f831e (127.29.78.131:33667), 25601140ec2a431380c5523a3ab49830 (127.29.78.129:36613)
I20260808 18:28:33.446712 31089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e78357a87b4ee39e4628fa8b3e4ec5" candidate_uuid: "8a4237a501ca4074a01bf24b05d55c09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "375798fdad754e199f171906607f831e"
I20260808 18:28:33.446760 30795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e78357a87b4ee39e4628fa8b3e4ec5" candidate_uuid: "8a4237a501ca4074a01bf24b05d55c09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25601140ec2a431380c5523a3ab49830"
I20260808 18:28:33.446830 31089 raft_consensus.cc:3037] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 375798fdad754e199f171906607f831e [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:33.446906 30795 raft_consensus.cc:3037] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 25601140ec2a431380c5523a3ab49830 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:33.447939 30795 raft_consensus.cc:2454] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 25601140ec2a431380c5523a3ab49830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a4237a501ca4074a01bf24b05d55c09 in term 1.
I20260808 18:28:33.448089 31089 raft_consensus.cc:2454] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 375798fdad754e199f171906607f831e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a4237a501ca4074a01bf24b05d55c09 in term 1.
I20260808 18:28:33.448251 30870 leader_election.cc:302] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25601140ec2a431380c5523a3ab49830, 8a4237a501ca4074a01bf24b05d55c09; no voters:
I20260808 18:28:33.448441 31169 raft_consensus.cc:2781] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:33.448668 31169 raft_consensus.cc:686] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09 [term 1 LEADER]: Becoming Leader. State: Replica: 8a4237a501ca4074a01bf24b05d55c09, State: Running, Role: LEADER
I20260808 18:28:33.449013 31169 consensus_queue.cc:237] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375798fdad754e199f171906607f831e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33667 } } peers { permanent_uuid: "8a4237a501ca4074a01bf24b05d55c09" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33611 } } peers { permanent_uuid: "25601140ec2a431380c5523a3ab49830" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36613 } }
I20260808 18:28:33.452368 30631 catalog_manager.cc:5495] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 8a4237a501ca4074a01bf24b05d55c09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a4237a501ca4074a01bf24b05d55c09 (127.29.78.130). New cstate: current_term: 1 leader_uuid: "8a4237a501ca4074a01bf24b05d55c09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "375798fdad754e199f171906607f831e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a4237a501ca4074a01bf24b05d55c09" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25601140ec2a431380c5523a3ab49830" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36613 } health_report { overall_health: UNKNOWN } } }
Aug 08 18:28:33 dist-test-slave-gp6n krb5kdc[30583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260808 18:28:33.477212 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:33.468540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43868 (local address 127.29.78.129:36613)
0808 18:28:33.468716 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:33.468720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:33.468888 (+ 168us) server_negotiation.cc:407] Connection header received
0808 18:28:33.469000 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:33.469002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:33.469042 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:33.469118 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:33.469869 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.470363 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.471243 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.471481 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.471560 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:33.473923 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:33.473933 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:33.473946 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:33.473969 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:33.475601 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:33.475979 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:33.475984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:33.475985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:33.476063 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:33.476410 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:33.476414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:33.476415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:33.476673 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:33.476827 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:33.477003 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 18:28:33.477098 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Aug 08 18:28:33 dist-test-slave-gp6n krb5kdc[30583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213713, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 18:28:33 dist-test-slave-gp6n krb5kdc[30583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213713, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260808 18:28:33.505347 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:33.496896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.29.78.129:36613)
0808 18:28:33.497085 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:33.497088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:33.497153 (+ 65us) server_negotiation.cc:407] Connection header received
0808 18:28:33.497264 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:33.497266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:33.497310 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:33.497381 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:33.498051 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.498709 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.499512 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.499752 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.499840 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:33.502161 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:33.502164 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:33.502166 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:33.502189 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:33.503740 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:33.504157 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:33.504160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:33.504162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:33.504239 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:33.504496 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:33.504500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:33.504501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:33.504670 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:33.504780 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:33.505024 (+ 244us) server_negotiation.cc:299] Negotiation successful
0808 18:28:33.505123 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
Aug 08 18:28:33 dist-test-slave-gp6n krb5kdc[30583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 18:28:33 dist-test-slave-gp6n krb5kdc[30583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260808 18:28:33.533027 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:33.524346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43878 (local address 127.29.78.129:36613)
0808 18:28:33.524519 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:33.524524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:33.524668 (+ 144us) server_negotiation.cc:407] Connection header received
0808 18:28:33.524851 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:33.524854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:33.524904 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:33.525002 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:33.525669 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.526180 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.527070 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:33.527293 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:33.527404 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:33.529988 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:33.529991 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:33.529993 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:33.530017 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:33.531497 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:33.531927 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:33.531930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:33.531931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:33.532025 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:33.532272 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:33.532274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:33.532275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:33.532449 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:33.532561 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:33.532787 (+ 226us) server_negotiation.cc:299] Negotiation successful
0808 18:28:33.532898 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260808 18:28:33.534538 30750 tablet_service.cc:2908] Rejecting scan request for tablet f7e78357a87b4ee39e4628fa8b3e4ec5: Uninitialized: clean time has not yet been initialized
I20260808 18:28:33.536703 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 30701
I20260808 18:28:33.544260 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 30852
W20260808 18:28:33.547922 31145 tablet.cc:2369] T f7e78357a87b4ee39e4628fa8b3e4ec5 P 375798fdad754e199f171906607f831e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:28:33.550992 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 30999
I20260808 18:28:33.557859 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 30598
2026-08-08T18:28:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2401 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:28:33 dist-test-slave-gp6n krb5kdc[31188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:28:33 dist-test-slave-gp6n krb5kdc[31188](info): set up 2 sockets
Aug 08 18:28:33 dist-test-slave-gp6n krb5kdc[31188](info): commencing operation
krb5kdc: starting...
W20260808 18:28:34.716972 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:28:34 dist-test-slave-gp6n krb5kdc[31188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:28:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:28:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:28:34.850286 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:33993
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:33993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:34.961295 31203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:34.961602 31203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:34.961683 31203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:34.965677 31203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:28:34.965762 31203 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 18:28:34.965795 31203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:34.965837 31203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:34.965870 31203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:28:34.965924 31203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:28:34.971164 31203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:33993
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:33993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:34.972191 31203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:34.973222 31203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:34.981699 31210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:34.981725 31208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:34.981757 31212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:34.981880 31203 server_base.cc:1034] running on GCE node
I20260808 18:28:34.982592 31203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:34.983669 31203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:34.984870 31203 hybrid_clock.cc:648] HybridClock initialized: now 1786213714984852 us; error 33 us; skew 500 ppm
Aug 08 18:28:34 dist-test-slave-gp6n krb5kdc[31188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213714, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:34.991865 31203 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:28:34.992770 31203 webserver.cc:460] Webserver started at http://127.29.78.190:44143/ using document root <none> and password file <none>
I20260808 18:28:34.993289 31203 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:34.993348 31203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:34.993512 31203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:34.995545 31203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "144cc6477b6d4c40ad5c304efe592b0b"
format_stamp: "Formatted at 2026-08-08 18:28:34 on dist-test-slave-gp6n"
server_key: "46f18a0b869aeb57ff9fe2573b6c45da"
server_key_iv: "51af9e45955321a97a5f2ff66cdf4f2f"
server_key_version: "encryptionkey@0"
I20260808 18:28:34.996075 31203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "144cc6477b6d4c40ad5c304efe592b0b"
format_stamp: "Formatted at 2026-08-08 18:28:34 on dist-test-slave-gp6n"
server_key: "46f18a0b869aeb57ff9fe2573b6c45da"
server_key_iv: "51af9e45955321a97a5f2ff66cdf4f2f"
server_key_version: "encryptionkey@0"
I20260808 18:28:34.999887 31203 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:28:35.002249 31220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:35.003590 31203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:28:35.003731 31203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "144cc6477b6d4c40ad5c304efe592b0b"
format_stamp: "Formatted at 2026-08-08 18:28:34 on dist-test-slave-gp6n"
server_key: "46f18a0b869aeb57ff9fe2573b6c45da"
server_key_iv: "51af9e45955321a97a5f2ff66cdf4f2f"
server_key_version: "encryptionkey@0"
I20260808 18:28:35.003847 31203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:35.027930 31203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:35.031378 31203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:35.031586 31203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:35.040548 31203 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:33993
I20260808 18:28:35.040576 31289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:33993 every 8 connection(s)
I20260808 18:28:35.041576 31203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:28:35.044457 31290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:35.047554 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 31203
I20260808 18:28:35.047650 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:28:35.047914 30010 external_mini_cluster.cc:1428] Setting key 6cdba021acb0c17dd5b5c87d11466ff0
I20260808 18:28:35.050652 31290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b: Bootstrap starting.
I20260808 18:28:35.053363 31290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:35.054239 31290 log.cc:824] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:35.056453 31290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b: No bootstrap required, opened a new log
Aug 08 18:28:35 dist-test-slave-gp6n krb5kdc[31188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:35.060994 31290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "144cc6477b6d4c40ad5c304efe592b0b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33993 } }
I20260808 18:28:35.061312 31290 raft_consensus.cc:374] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:35.061383 31290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 144cc6477b6d4c40ad5c304efe592b0b, State: Initialized, Role: FOLLOWER
I20260808 18:28:35.061905 31290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "144cc6477b6d4c40ad5c304efe592b0b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33993 } }
I20260808 18:28:35.062076 31290 raft_consensus.cc:388] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:28:35.062150 31290 raft_consensus.cc:482] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:28:35.062263 31290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:35.063362 31290 raft_consensus.cc:504] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "144cc6477b6d4c40ad5c304efe592b0b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33993 } }
I20260808 18:28:35.063705 31290 leader_election.cc:302] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 144cc6477b6d4c40ad5c304efe592b0b; no voters:
I20260808 18:28:35.063962 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.049413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44446 (local address 127.29.78.190:33993)
0808 18:28:35.049835 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:35.049854 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:35.049898 (+ 44us) server_negotiation.cc:407] Connection header received
0808 18:28:35.050748 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:35.050771 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:35.051182 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:35.051544 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:35.052205 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.053718 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.055006 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.055736 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.055908 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.058732 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:35.058739 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:35.058755 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:35.058790 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:35.060716 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:35.061150 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:35.061157 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:35.061162 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:35.061312 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:35.061601 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:35.061604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:35.061606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:35.062118 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:35.062329 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:35.062642 (+ 313us) server_negotiation.cc:299] Negotiation successful
0808 18:28:35.062930 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":112,"threads_started":1}
I20260808 18:28:35.064042 31290 leader_election.cc:288] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:28:35.064172 31295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:35.064378 31295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b [term 1 LEADER]: Becoming Leader. State: Replica: 144cc6477b6d4c40ad5c304efe592b0b, State: Running, Role: LEADER
I20260808 18:28:35.064678 31295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "144cc6477b6d4c40ad5c304efe592b0b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33993 } }
I20260808 18:28:35.065294 31290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:28:35.066324 31296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "144cc6477b6d4c40ad5c304efe592b0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "144cc6477b6d4c40ad5c304efe592b0b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33993 } } }
I20260808 18:28:35.066471 31296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:35.066787 31297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 144cc6477b6d4c40ad5c304efe592b0b. Latest consensus state: current_term: 1 leader_uuid: "144cc6477b6d4c40ad5c304efe592b0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "144cc6477b6d4c40ad5c304efe592b0b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33993 } } }
I20260808 18:28:35.066854 31305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:28:35.066879 31297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:35.069967 31305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:28:35.075731 31305 catalog_manager.cc:1347] Generated new cluster ID: 71b809c490e84890ae9e3186c9021278
I20260808 18:28:35.075819 31305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:28:35.085759 31305 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:28:35.087044 31305 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:28:35.109009 31305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b: Generated new TSK 0
I20260808 18:28:35.109774 31305 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260808 18:28:35.140892 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:33993
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:35.277748 31318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:35.278013 31318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:35.278059 31318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:35.281976 31318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:35.282037 31318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:35.282074 31318 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 18:28:35.282105 31318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:28:35.287405 31318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.78.190:33993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:35.288654 31318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:35.289757 31318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:35.300086 31323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:35.300108 31324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:35.300086 31326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:35.300510 31318 server_base.cc:1034] running on GCE node
I20260808 18:28:35.300948 31318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:35.301645 31318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:35.302932 31318 hybrid_clock.cc:648] HybridClock initialized: now 1786213715302903 us; error 50 us; skew 500 ppm
Aug 08 18:28:35 dist-test-slave-gp6n krb5kdc[31188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213715, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:35.305725 31318 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:28:35.306695 31318 webserver.cc:460] Webserver started at http://127.29.78.129:42265/ using document root <none> and password file <none>
I20260808 18:28:35.307308 31318 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:35.307400 31318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:35.307637 31318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:35.309765 31318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "25886af7e7884fad86bbb0878c075c7c"
format_stamp: "Formatted at 2026-08-08 18:28:35 on dist-test-slave-gp6n"
server_key: "febfddbd2da9b543bd3ecb15b5b16b26"
server_key_iv: "1da40ea1765f317028ebf1f17084da28"
server_key_version: "encryptionkey@0"
I20260808 18:28:35.310338 31318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "25886af7e7884fad86bbb0878c075c7c"
format_stamp: "Formatted at 2026-08-08 18:28:35 on dist-test-slave-gp6n"
server_key: "febfddbd2da9b543bd3ecb15b5b16b26"
server_key_iv: "1da40ea1765f317028ebf1f17084da28"
server_key_version: "encryptionkey@0"
I20260808 18:28:35.313927 31318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:28:35.316316 31333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:35.317762 31318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 18:28:35.317898 31318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "25886af7e7884fad86bbb0878c075c7c"
format_stamp: "Formatted at 2026-08-08 18:28:35 on dist-test-slave-gp6n"
server_key: "febfddbd2da9b543bd3ecb15b5b16b26"
server_key_iv: "1da40ea1765f317028ebf1f17084da28"
server_key_version: "encryptionkey@0"
I20260808 18:28:35.318001 31318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:35.338974 31318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:35.342113 31318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:35.342335 31318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:35.343904 31318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:35.343955 31318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:35.343997 31318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:35.344054 31318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 18:28:35 dist-test-slave-gp6n krb5kdc[31188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213715, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:35.354811 31318 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:35115
I20260808 18:28:35.354892 31463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:35115 every 8 connection(s)
I20260808 18:28:35.355998 31318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:28:35.358453 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 31318
I20260808 18:28:35.358577 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:28:35.358625 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.344619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:54751 (local address 127.29.78.190:33993)
0808 18:28:35.344814 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:35.344821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:35.346023 (+ 1202us) server_negotiation.cc:407] Connection header received
0808 18:28:35.348282 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:35.348286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:35.348345 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:35.348450 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:35.349613 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.350149 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.351503 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.351858 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.351975 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.354911 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:35.354915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:35.354918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:35.354947 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:35.356583 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:35.357162 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:35.357165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:35.357166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:35.357245 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:35.357508 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:35.357510 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:35.357511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:35.357698 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:35.357772 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:35.358282 (+ 510us) server_negotiation.cc:299] Negotiation successful
0808 18:28:35.358439 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 18:28:35.358886 30010 external_mini_cluster.cc:1428] Setting key d495f79707839f699714e13f9f9b410c
I20260808 18:28:35.359489 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.345414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:33993 (local address 127.29.78.129:54751)
0808 18:28:35.345853 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:35.345901 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:35.348053 (+ 2152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:35.348641 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:35.348653 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:35.348999 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:35.349462 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:35.349476 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.350277 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.350281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:35.351375 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:35.351384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.352010 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.352014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:35.352199 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.352803 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:35.352829 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:35.354714 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:35.356733 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:35.356740 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:35.356758 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:35.357053 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:35.357305 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:35.357307 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:35.357308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:35.357427 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:35.357791 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:35.357796 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:35.358030 (+ 234us) client_negotiation.cc:769] Sending connection context
0808 18:28:35.358179 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 18:28:35.358489 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":126,"threads_started":1}
I20260808 18:28:35.360976 31464 heartbeater.cc:347] Connected to a master server at 127.29.78.190:33993
I20260808 18:28:35.361133 31464 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:35.361686 31464 heartbeater.cc:511] Master 127.29.78.190:33993 requested a full tablet report, sending...
I20260808 18:28:35.363389 31238 ts_manager.cc:194] Registered new tserver with Master: 25886af7e7884fad86bbb0878c075c7c (127.29.78.129:35115)
I20260808 18:28:35.365105 31238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:54751
I20260808 18:28:35.370687 31471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.363135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:33993 (local address 127.29.78.129:48159)
0808 18:28:35.363400 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:35.363413 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:35.363523 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:35.363806 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:35.363809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:35.364012 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:35.364107 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:35.364113 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.365071 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.365074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:35.365879 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:35.365889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.366459 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.366463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:35.366556 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.367075 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:35.367086 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:35.367405 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:35.369101 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:35.369105 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:35.369107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:35.369321 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:35.369622 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:35.369626 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:35.369628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:35.369719 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:35.370095 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:35.370098 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:35.370196 (+ 98us) client_negotiation.cc:769] Sending connection context
0808 18:28:35.370303 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 18:28:35.370486 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":104,"threads_started":1}
I20260808 18:28:35.370680 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.363189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:48159 (local address 127.29.78.190:33993)
0808 18:28:35.363302 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:35.363306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:35.363555 (+ 249us) server_negotiation.cc:407] Connection header received
0808 18:28:35.363644 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:35.363647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:35.363694 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:35.363784 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:35.364221 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.364939 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.366074 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.366305 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.366455 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.367548 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:35.367551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:35.367554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:35.367577 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:35.368974 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:35.369436 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:35.369440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:35.369442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:35.369529 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:35.369812 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:35.369817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:35.369819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:35.369993 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:35.370104 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:35.370368 (+ 264us) server_negotiation.cc:299] Negotiation successful
0808 18:28:35.370510 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":37}
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260808 18:28:35.389948 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:33993
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:35.499888 31473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:35.500134 31473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:35.500178 31473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:35.504163 31473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:35.504217 31473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:35.504287 31473 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 18:28:35.504372 31473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:28:35.509742 31473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.78.190:33993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:35.510778 31473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:35.511761 31473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:35.521561 31479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:35.521694 31473 server_base.cc:1034] running on GCE node
W20260808 18:28:35.521567 31478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:35.521595 31481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:35.522258 31473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:35.522961 31473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:35.524122 31473 hybrid_clock.cc:648] HybridClock initialized: now 1786213715524106 us; error 27 us; skew 500 ppm
Aug 08 18:28:35 dist-test-slave-gp6n krb5kdc[31188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213715, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:35.526793 31473 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:28:35.527679 31473 webserver.cc:460] Webserver started at http://127.29.78.130:46191/ using document root <none> and password file <none>
I20260808 18:28:35.528223 31473 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:35.528282 31473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:35.528491 31473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:35.530527 31473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac"
format_stamp: "Formatted at 2026-08-08 18:28:35 on dist-test-slave-gp6n"
server_key: "cff34200f0ea7045f3c0670a0b82bc76"
server_key_iv: "5c849801989f21ed8ce47631e42dbd8f"
server_key_version: "encryptionkey@0"
I20260808 18:28:35.531081 31473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac"
format_stamp: "Formatted at 2026-08-08 18:28:35 on dist-test-slave-gp6n"
server_key: "cff34200f0ea7045f3c0670a0b82bc76"
server_key_iv: "5c849801989f21ed8ce47631e42dbd8f"
server_key_version: "encryptionkey@0"
I20260808 18:28:35.534734 31473 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:28:35.537046 31489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:35.538211 31473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:28:35.538327 31473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac"
format_stamp: "Formatted at 2026-08-08 18:28:35 on dist-test-slave-gp6n"
server_key: "cff34200f0ea7045f3c0670a0b82bc76"
server_key_iv: "5c849801989f21ed8ce47631e42dbd8f"
server_key_version: "encryptionkey@0"
I20260808 18:28:35.538486 31473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:35.555765 31473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:35.558570 31473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:35.558761 31473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:35.560125 31473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:35.560199 31473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:35.560269 31473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:35.560312 31473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 18:28:35 dist-test-slave-gp6n krb5kdc[31188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213715, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:35.571689 31473 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:39113
I20260808 18:28:35.571811 31623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:39113 every 8 connection(s)
I20260808 18:28:35.572564 31473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:28:35.574781 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.560976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:47187 (local address 127.29.78.190:33993)
0808 18:28:35.561172 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:35.561176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:35.562245 (+ 1069us) server_negotiation.cc:407] Connection header received
0808 18:28:35.563551 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:35.563557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:35.563635 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:35.563747 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:35.565267 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.566236 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.567808 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.568220 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.568342 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.571050 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:35.571057 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:35.571062 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:35.571094 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:35.572780 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:35.573398 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:35.573402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:35.573404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:35.573491 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:35.573820 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:35.573823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:35.573824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:35.573991 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:35.574075 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:35.574567 (+ 492us) server_negotiation.cc:299] Negotiation successful
0808 18:28:35.574663 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 18:28:35.575543 31501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.561615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:33993 (local address 127.29.78.130:47187)
0808 18:28:35.562079 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:35.562121 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:35.563249 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:35.563961 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:35.563972 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:35.564481 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:35.565109 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:35.565132 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.566353 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.566357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:35.567704 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:35.567713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.568415 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.568420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:35.568628 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.569226 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:35.569242 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:35.570874 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:35.572891 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:35.572896 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:35.572909 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:35.573278 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:35.573584 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:35.573587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:35.573589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:35.573731 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:35.574088 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:35.574092 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:35.574298 (+ 206us) client_negotiation.cc:769] Sending connection context
0808 18:28:35.574498 (+ 200us) client_negotiation.cc:240] Negotiation successful
0808 18:28:35.574748 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":127,"threads_started":1}
I20260808 18:28:35.576565 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 31473
I20260808 18:28:35.576680 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:28:35.576853 31624 heartbeater.cc:347] Connected to a master server at 127.29.78.190:33993
I20260808 18:28:35.576974 30010 external_mini_cluster.cc:1428] Setting key e5d9682adac05a6fd9ea4d2021a8965c
I20260808 18:28:35.576984 31624 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:35.577512 31624 heartbeater.cc:511] Master 127.29.78.190:33993 requested a full tablet report, sending...
I20260808 18:28:35.578514 31238 ts_manager.cc:194] Registered new tserver with Master: 8e53a55ef9bf491d9ac92ffd8dfb0fac (127.29.78.130:39113)
I20260808 18:28:35.579114 31238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:47187
I20260808 18:28:35.586319 31633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.578954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:33993 (local address 127.29.78.130:55187)
0808 18:28:35.579251 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:35.579266 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:35.579349 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:35.579662 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:35.579668 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:35.579872 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:35.579991 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:35.579998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.580957 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.580960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:35.581607 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:35.581615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.582051 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.582055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:35.582146 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.582694 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:35.582713 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:35.583094 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:35.584783 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:35.584786 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:35.584788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:35.584986 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:35.585275 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:35.585277 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:35.585279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:35.585366 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:35.585795 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:35.585798 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:35.585889 (+ 91us) client_negotiation.cc:769] Sending connection context
0808 18:28:35.586009 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 18:28:35.586142 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":2,"thread_start_us":100,"threads_started":1}
I20260808 18:28:35.586316 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.579303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:55187 (local address 127.29.78.190:33993)
0808 18:28:35.579427 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:35.579434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:35.579449 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:28:35.579489 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:35.579494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:35.579544 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:35.579628 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:35.580127 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.580831 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.581722 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.581940 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.582048 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.583220 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:35.583224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:35.583227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:35.583249 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:35.584654 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:35.585088 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:35.585092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:35.585094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:35.585174 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:35.585457 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:35.585462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:35.585464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:35.585670 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:35.585812 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:35.586077 (+ 265us) server_negotiation.cc:299] Negotiation successful
0808 18:28:35.586182 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260808 18:28:35.605343 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:33993
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:35.714790 31636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:35.715070 31636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:35.715171 31636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:35.719278 31636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:35.719352 31636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:35.719419 31636 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 18:28:35.719496 31636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:28:35.724727 31636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.78.190:33993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:35.725692 31636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:35.726732 31636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:35.736104 31643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:35.736110 31648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:35.736110 31644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:35.736383 31636 server_base.cc:1034] running on GCE node
I20260808 18:28:35.736814 31636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:35.737459 31636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:35.738631 31636 hybrid_clock.cc:648] HybridClock initialized: now 1786213715738597 us; error 45 us; skew 500 ppm
Aug 08 18:28:35 dist-test-slave-gp6n krb5kdc[31188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213715, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:35.741304 31636 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:28:35.742192 31636 webserver.cc:460] Webserver started at http://127.29.78.131:41809/ using document root <none> and password file <none>
I20260808 18:28:35.742758 31636 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:35.742818 31636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:35.742977 31636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:35.744874 31636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "bd5c10e848e34f599161b0a6f800ccde"
format_stamp: "Formatted at 2026-08-08 18:28:35 on dist-test-slave-gp6n"
server_key: "fb838bdb0af16572b5c0b0ba231380ee"
server_key_iv: "f8c94da4dd1429690c52fde992c18697"
server_key_version: "encryptionkey@0"
I20260808 18:28:35.745384 31636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "bd5c10e848e34f599161b0a6f800ccde"
format_stamp: "Formatted at 2026-08-08 18:28:35 on dist-test-slave-gp6n"
server_key: "fb838bdb0af16572b5c0b0ba231380ee"
server_key_iv: "f8c94da4dd1429690c52fde992c18697"
server_key_version: "encryptionkey@0"
I20260808 18:28:35.749032 31636 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 18:28:35.751317 31655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:35.752452 31636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:28:35.752576 31636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "bd5c10e848e34f599161b0a6f800ccde"
format_stamp: "Formatted at 2026-08-08 18:28:35 on dist-test-slave-gp6n"
server_key: "fb838bdb0af16572b5c0b0ba231380ee"
server_key_iv: "f8c94da4dd1429690c52fde992c18697"
server_key_version: "encryptionkey@0"
I20260808 18:28:35.752681 31636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:35.766387 31636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:35.769079 31636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:35.769270 31636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:35.770681 31636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:35.770756 31636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:35.770828 31636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:35.770872 31636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 18:28:35 dist-test-slave-gp6n krb5kdc[31188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213715, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:35.781980 31636 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:40189
I20260808 18:28:35.782147 31784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:40189 every 8 connection(s)
I20260808 18:28:35.783033 31636 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:28:35.784735 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.771370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:43121 (local address 127.29.78.190:33993)
0808 18:28:35.771534 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:35.771539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:35.772800 (+ 1261us) server_negotiation.cc:407] Connection header received
0808 18:28:35.774000 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:35.774005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:35.774083 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:35.774189 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:35.775426 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.775965 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.777231 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.777543 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.777608 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.780654 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:35.780657 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:35.780660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:35.780694 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:35.782575 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:35.783155 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:35.783158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:35.783160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:35.783256 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:35.783658 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:35.783661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:35.783663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:35.783854 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:35.783963 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:35.784520 (+ 557us) server_negotiation.cc:299] Negotiation successful
0808 18:28:35.784620 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 18:28:35.785442 31665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.772187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:33993 (local address 127.29.78.131:43121)
0808 18:28:35.772629 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:35.772684 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:35.773754 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:35.774455 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:35.774464 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:35.774846 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:35.775279 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:35.775292 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.776094 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.776099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:35.777043 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:35.777052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.777776 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.777778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:35.777908 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.778482 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:35.778509 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:35.780455 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:35.782682 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:35.782691 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:35.782707 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:35.783046 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:35.783355 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:35.783359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:35.783362 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:35.783545 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:35.783963 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:35.783970 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:35.784247 (+ 277us) client_negotiation.cc:769] Sending connection context
0808 18:28:35.784453 (+ 206us) client_negotiation.cc:240] Negotiation successful
0808 18:28:35.784696 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":131,"threads_started":1}
I20260808 18:28:35.786669 31785 heartbeater.cc:347] Connected to a master server at 127.29.78.190:33993
I20260808 18:28:35.786806 31785 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:35.787262 31785 heartbeater.cc:511] Master 127.29.78.190:33993 requested a full tablet report, sending...
I20260808 18:28:35.788270 31237 ts_manager.cc:194] Registered new tserver with Master: bd5c10e848e34f599161b0a6f800ccde (127.29.78.131:40189)
I20260808 18:28:35.789040 31237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:43121
I20260808 18:28:35.791815 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 31636
I20260808 18:28:35.791955 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:28:35.792341 30010 external_mini_cluster.cc:1428] Setting key d1a9a1f120db4f589fea9a900939aac4
I20260808 18:28:35.794445 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:28:35.795938 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.788768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:33215 (local address 127.29.78.190:33993)
0808 18:28:35.788891 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:35.788895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:35.789055 (+ 160us) server_negotiation.cc:407] Connection header received
0808 18:28:35.789126 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:35.789129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:35.789167 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:35.789250 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:35.789749 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.790225 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.791225 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.791426 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.791513 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.792784 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:35.792787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:35.792790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:35.792812 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:35.794329 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:35.794776 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:35.794781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:35.794783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:35.794865 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:35.795159 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:35.795164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:35.795166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:35.795332 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:35.795440 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:35.795698 (+ 258us) server_negotiation.cc:299] Negotiation successful
0808 18:28:35.795803 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 18:28:35.795950 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.788645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:33993 (local address 127.29.78.131:33215)
0808 18:28:35.788974 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:35.788986 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:35.789064 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:35.789284 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:35.789286 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:35.789491 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:35.789635 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:35.789642 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.790352 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.790356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:35.791126 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:35.791135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.791552 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.791555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:35.791655 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.792222 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:35.792237 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:35.792636 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:35.794446 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:35.794452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:35.794455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:35.794676 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:35.794964 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:35.794967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:35.794969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:35.795068 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:35.795437 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:35.795440 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:35.795535 (+ 95us) client_negotiation.cc:769] Sending connection context
0808 18:28:35.795642 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 18:28:35.795775 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":121,"threads_started":1}
Aug 08 18:28:35 dist-test-slave-gp6n krb5kdc[31188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213715, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 18:28:35 dist-test-slave-gp6n krb5kdc[31188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213715, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:35.820680 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.812706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44452 (local address 127.29.78.190:33993)
0808 18:28:35.812828 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:35.812832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:35.812842 (+ 10us) server_negotiation.cc:407] Connection header received
0808 18:28:35.812877 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:35.812880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:35.812916 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:35.813009 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:35.813717 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.814234 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.815133 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.815350 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.815457 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.817912 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:35.817916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:35.817918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:35.817941 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:35.819357 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:35.819736 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:35.819738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:35.819739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:35.819812 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:35.820032 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:35.820034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:35.820035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:35.820191 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:35.820285 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:35.820462 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 18:28:35.820560 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 18:28:35.823760 31237 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:28:35.826798 31237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:28:35.841866 31807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.835061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54506 (local address 127.29.78.129:35115)
0808 18:28:35.836330 (+ 1269us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:35.836347 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:35.836369 (+ 22us) server_negotiation.cc:407] Connection header received
0808 18:28:35.836431 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:35.836435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:35.836610 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:35.836771 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:35.837182 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.838482 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.840438 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.841083 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.841204 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.841445 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:35.841581 (+ 136us) server_negotiation.cc:299] Negotiation successful
0808 18:28:35.841688 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1161,"thread_start_us":89,"threads_started":1}
I20260808 18:28:35.841866 31805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.834890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:35115 (local address 127.0.0.1:54506)
0808 18:28:35.835716 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:35.835777 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:35.835913 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:35.836779 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:35.836785 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:35.836809 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:35.836970 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:35.836978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.838627 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.838634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:35.840278 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:35.840286 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.841254 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.841258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:35.841372 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.841412 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 18:28:35.841536 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 18:28:35.841702 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":206,"thread_start_us":85,"threads_started":1}
I20260808 18:28:35.841908 31808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.835273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:39113 (local address 127.0.0.1:40900)
0808 18:28:35.836156 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:35.836168 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:35.836248 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:35.836934 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:35.836939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:35.836957 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:35.837255 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:35.837273 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.838658 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.838661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:35.840015 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:35.840027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.841307 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.841311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:35.841436 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.841464 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 18:28:35.841520 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 18:28:35.841710 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":326,"thread_start_us":97,"threads_started":1}
I20260808 18:28:35.842099 31806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.835004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:40189 (local address 127.0.0.1:38064)
0808 18:28:35.835963 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:35.835983 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:35.836072 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:35.836954 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:35.836957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:35.836969 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:35.837163 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:35.837171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.838324 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.838331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:35.840279 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:35.840291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.841683 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.841687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:35.841790 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.841813 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 18:28:35.841888 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 18:28:35.841966 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":467,"thread_start_us":89,"threads_started":1}
I20260808 18:28:35.842751 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.835713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38064 (local address 127.29.78.131:40189)
0808 18:28:35.836512 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:35.836531 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:35.836549 (+ 18us) server_negotiation.cc:407] Connection header received
0808 18:28:35.836609 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:35.836612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:35.836796 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:35.836984 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:35.837315 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.838190 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.840458 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.841319 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.841437 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.841614 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:35.842511 (+ 897us) server_negotiation.cc:299] Negotiation successful
0808 18:28:35.842612 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":110,"threads_started":1}
I20260808 18:28:35.842937 31809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.835560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.29.78.130:39113)
0808 18:28:35.836195 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:35.836210 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:35.836233 (+ 23us) server_negotiation.cc:407] Connection header received
0808 18:28:35.836388 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:35.836396 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:35.836610 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:35.836771 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:35.837407 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.838545 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.840156 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.841099 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.842354 (+ 1255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.842601 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:35.842661 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 18:28:35.842770 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":116,"threads_started":1}
I20260808 18:28:35.845316 31710 tablet_service.cc:1467] Processing CreateTablet for tablet 17974244c9d94bd2b2dfcf4490f821b2 (DEFAULT_TABLE table=test-table [id=c85b08913617490dabedffe30ef64c25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:35.845366 31551 tablet_service.cc:1467] Processing CreateTablet for tablet 17974244c9d94bd2b2dfcf4490f821b2 (DEFAULT_TABLE table=test-table [id=c85b08913617490dabedffe30ef64c25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:35.845371 31390 tablet_service.cc:1467] Processing CreateTablet for tablet 17974244c9d94bd2b2dfcf4490f821b2 (DEFAULT_TABLE table=test-table [id=c85b08913617490dabedffe30ef64c25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:35.846457 31551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17974244c9d94bd2b2dfcf4490f821b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:35.846457 31390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17974244c9d94bd2b2dfcf4490f821b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:35.846524 31710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17974244c9d94bd2b2dfcf4490f821b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:35.853916 31811 tablet_bootstrap.cc:492] T 17974244c9d94bd2b2dfcf4490f821b2 P bd5c10e848e34f599161b0a6f800ccde: Bootstrap starting.
I20260808 18:28:35.855420 31813 tablet_bootstrap.cc:492] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: Bootstrap starting.
I20260808 18:28:35.856606 31811 tablet_bootstrap.cc:654] T 17974244c9d94bd2b2dfcf4490f821b2 P bd5c10e848e34f599161b0a6f800ccde: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:35.857626 31811 log.cc:824] T 17974244c9d94bd2b2dfcf4490f821b2 P bd5c10e848e34f599161b0a6f800ccde: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:35.857681 31813 tablet_bootstrap.cc:654] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:35.858523 31812 tablet_bootstrap.cc:492] T 17974244c9d94bd2b2dfcf4490f821b2 P 25886af7e7884fad86bbb0878c075c7c: Bootstrap starting.
I20260808 18:28:35.858603 31813 log.cc:824] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:35.860122 31811 tablet_bootstrap.cc:492] T 17974244c9d94bd2b2dfcf4490f821b2 P bd5c10e848e34f599161b0a6f800ccde: No bootstrap required, opened a new log
I20260808 18:28:35.860352 31811 ts_tablet_manager.cc:1397] T 17974244c9d94bd2b2dfcf4490f821b2 P bd5c10e848e34f599161b0a6f800ccde: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 18:28:35.860699 31812 tablet_bootstrap.cc:654] T 17974244c9d94bd2b2dfcf4490f821b2 P 25886af7e7884fad86bbb0878c075c7c: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:35.860792 31813 tablet_bootstrap.cc:492] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: No bootstrap required, opened a new log
I20260808 18:28:35.861019 31813 ts_tablet_manager.cc:1397] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:28:35.861594 31812 log.cc:824] T 17974244c9d94bd2b2dfcf4490f821b2 P 25886af7e7884fad86bbb0878c075c7c: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:35.863745 31812 tablet_bootstrap.cc:492] T 17974244c9d94bd2b2dfcf4490f821b2 P 25886af7e7884fad86bbb0878c075c7c: No bootstrap required, opened a new log
I20260808 18:28:35.863945 31812 ts_tablet_manager.cc:1397] T 17974244c9d94bd2b2dfcf4490f821b2 P 25886af7e7884fad86bbb0878c075c7c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:28:35.864117 31811 raft_consensus.cc:348] T 17974244c9d94bd2b2dfcf4490f821b2 P bd5c10e848e34f599161b0a6f800ccde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } }
I20260808 18:28:35.864408 31811 raft_consensus.cc:374] T 17974244c9d94bd2b2dfcf4490f821b2 P bd5c10e848e34f599161b0a6f800ccde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:35.864467 31811 raft_consensus.cc:729] T 17974244c9d94bd2b2dfcf4490f821b2 P bd5c10e848e34f599161b0a6f800ccde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd5c10e848e34f599161b0a6f800ccde, State: Initialized, Role: FOLLOWER
I20260808 18:28:35.864984 31811 consensus_queue.cc:260] T 17974244c9d94bd2b2dfcf4490f821b2 P bd5c10e848e34f599161b0a6f800ccde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } }
I20260808 18:28:35.865130 31813 raft_consensus.cc:348] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } }
I20260808 18:28:35.865413 31813 raft_consensus.cc:374] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:35.865474 31813 raft_consensus.cc:729] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e53a55ef9bf491d9ac92ffd8dfb0fac, State: Initialized, Role: FOLLOWER
I20260808 18:28:35.865896 31813 consensus_queue.cc:260] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } }
I20260808 18:28:35.866348 31811 ts_tablet_manager.cc:1428] T 17974244c9d94bd2b2dfcf4490f821b2 P bd5c10e848e34f599161b0a6f800ccde: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 18:28:35.866701 31785 heartbeater.cc:503] Master 127.29.78.190:33993 was elected leader, sending a full tablet report...
I20260808 18:28:35.866701 31624 heartbeater.cc:503] Master 127.29.78.190:33993 was elected leader, sending a full tablet report...
I20260808 18:28:35.868028 31813 ts_tablet_manager.cc:1428] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 18:28:35.869128 31812 raft_consensus.cc:348] T 17974244c9d94bd2b2dfcf4490f821b2 P 25886af7e7884fad86bbb0878c075c7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } }
I20260808 18:28:35.869360 31812 raft_consensus.cc:374] T 17974244c9d94bd2b2dfcf4490f821b2 P 25886af7e7884fad86bbb0878c075c7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:35.869426 31812 raft_consensus.cc:729] T 17974244c9d94bd2b2dfcf4490f821b2 P 25886af7e7884fad86bbb0878c075c7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25886af7e7884fad86bbb0878c075c7c, State: Initialized, Role: FOLLOWER
I20260808 18:28:35.869834 31812 consensus_queue.cc:260] T 17974244c9d94bd2b2dfcf4490f821b2 P 25886af7e7884fad86bbb0878c075c7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } }
I20260808 18:28:35.870555 31464 heartbeater.cc:503] Master 127.29.78.190:33993 was elected leader, sending a full tablet report...
I20260808 18:28:35.871431 31812 ts_tablet_manager.cc:1428] T 17974244c9d94bd2b2dfcf4490f821b2 P 25886af7e7884fad86bbb0878c075c7c: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 18:28:35.920157 31818 raft_consensus.cc:482] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:28:35.920370 31818 raft_consensus.cc:504] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } }
I20260808 18:28:35.921536 31818 leader_election.cc:288] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25886af7e7884fad86bbb0878c075c7c (127.29.78.129:35115), bd5c10e848e34f599161b0a6f800ccde (127.29.78.131:40189)
I20260808 18:28:35.925316 31807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.921952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:56315 (local address 127.29.78.129:35115)
0808 18:28:35.922101 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:35.922106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:35.922126 (+ 20us) server_negotiation.cc:407] Connection header received
0808 18:28:35.922172 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:35.922174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:35.922212 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:35.922309 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:35.922649 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.923352 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.924366 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.924763 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.924890 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.924944 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:35.925133 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 18:28:35.925200 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 18:28:35.925316 31501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.921769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:35115 (local address 127.29.78.130:56315)
0808 18:28:35.921902 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:35.921917 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:35.922008 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:35.922342 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:35.922346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:35.922367 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:35.922510 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:35.922516 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.923456 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.923462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:35.924235 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:35.924243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.924882 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.924885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:35.925000 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.925027 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 18:28:35.925115 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 18:28:35.925185 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260808 18:28:35.925909 31411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17974244c9d94bd2b2dfcf4490f821b2" candidate_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25886af7e7884fad86bbb0878c075c7c" is_pre_election: true
I20260808 18:28:35.926237 31411 raft_consensus.cc:2454] T 17974244c9d94bd2b2dfcf4490f821b2 P 25886af7e7884fad86bbb0878c075c7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e53a55ef9bf491d9ac92ffd8dfb0fac in term 0.
I20260808 18:28:35.926312 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.922033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:45673 (local address 127.29.78.131:40189)
0808 18:28:35.922183 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:35.922190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:35.922651 (+ 461us) server_negotiation.cc:407] Connection header received
0808 18:28:35.922756 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:35.922758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:35.922795 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:35.922853 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:35.923139 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.923621 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.925183 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.925742 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.925876 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.925921 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:35.926142 (+ 221us) server_negotiation.cc:299] Negotiation successful
0808 18:28:35.926208 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 18:28:35.926326 31820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.921785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:40189 (local address 127.29.78.130:45673)
0808 18:28:35.922544 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:35.922557 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:35.922675 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:35.922887 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:35.922889 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:35.922907 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:35.923036 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:35.923042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.923738 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.923743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:35.925052 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:35.925061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.925878 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.925881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:35.925984 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.926012 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 18:28:35.926112 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 18:28:35.926181 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":340,"thread_start_us":171,"threads_started":1}
I20260808 18:28:35.926846 31494 leader_election.cc:302] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25886af7e7884fad86bbb0878c075c7c, 8e53a55ef9bf491d9ac92ffd8dfb0fac; no voters:
I20260808 18:28:35.926909 31730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17974244c9d94bd2b2dfcf4490f821b2" candidate_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd5c10e848e34f599161b0a6f800ccde" is_pre_election: true
I20260808 18:28:35.927114 31818 raft_consensus.cc:2781] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:28:35.927201 31818 raft_consensus.cc:482] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:28:35.927213 31730 raft_consensus.cc:2454] T 17974244c9d94bd2b2dfcf4490f821b2 P bd5c10e848e34f599161b0a6f800ccde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e53a55ef9bf491d9ac92ffd8dfb0fac in term 0.
I20260808 18:28:35.927271 31818 raft_consensus.cc:3037] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:35.928440 31818 raft_consensus.cc:504] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } }
I20260808 18:28:35.928824 31818 leader_election.cc:288] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [CANDIDATE]: Term 1 election: Requested vote from peers 25886af7e7884fad86bbb0878c075c7c (127.29.78.129:35115), bd5c10e848e34f599161b0a6f800ccde (127.29.78.131:40189)
I20260808 18:28:35.929145 31411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17974244c9d94bd2b2dfcf4490f821b2" candidate_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25886af7e7884fad86bbb0878c075c7c"
I20260808 18:28:35.929163 31730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17974244c9d94bd2b2dfcf4490f821b2" candidate_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd5c10e848e34f599161b0a6f800ccde"
I20260808 18:28:35.929268 31730 raft_consensus.cc:3037] T 17974244c9d94bd2b2dfcf4490f821b2 P bd5c10e848e34f599161b0a6f800ccde [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:35.929287 31411 raft_consensus.cc:3037] T 17974244c9d94bd2b2dfcf4490f821b2 P 25886af7e7884fad86bbb0878c075c7c [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:35.930377 31730 raft_consensus.cc:2454] T 17974244c9d94bd2b2dfcf4490f821b2 P bd5c10e848e34f599161b0a6f800ccde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e53a55ef9bf491d9ac92ffd8dfb0fac in term 1.
I20260808 18:28:35.930734 31495 leader_election.cc:302] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e53a55ef9bf491d9ac92ffd8dfb0fac, bd5c10e848e34f599161b0a6f800ccde; no voters:
I20260808 18:28:35.930789 31411 raft_consensus.cc:2454] T 17974244c9d94bd2b2dfcf4490f821b2 P 25886af7e7884fad86bbb0878c075c7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e53a55ef9bf491d9ac92ffd8dfb0fac in term 1.
I20260808 18:28:35.930975 31818 raft_consensus.cc:2781] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:35.931205 31818 raft_consensus.cc:686] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 1 LEADER]: Becoming Leader. State: Replica: 8e53a55ef9bf491d9ac92ffd8dfb0fac, State: Running, Role: LEADER
I20260808 18:28:35.931535 31818 consensus_queue.cc:237] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } }
I20260808 18:28:35.934863 31235 catalog_manager.cc:5495] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e53a55ef9bf491d9ac92ffd8dfb0fac (127.29.78.130). New cstate: current_term: 1 leader_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } health_report { overall_health: HEALTHY } } }
I20260808 18:28:35.970103 31809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:35.966504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.29.78.130:39113)
0808 18:28:35.966656 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:35.966663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:35.966683 (+ 20us) server_negotiation.cc:407] Connection header received
0808 18:28:35.966931 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:35.966934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:35.967005 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:35.967130 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:28:35.967437 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.967942 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.968770 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:35.968980 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:35.969063 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:35.969335 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:28:35.969754 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:28:35.969839 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:35.969936 (+ 97us) server_negotiation.cc:299] Negotiation successful
0808 18:28:35.969985 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 18:28:35.975888 31411 raft_consensus.cc:1261] T 17974244c9d94bd2b2dfcf4490f821b2 P 25886af7e7884fad86bbb0878c075c7c [term 1 FOLLOWER]: Refusing update from remote peer 8e53a55ef9bf491d9ac92ffd8dfb0fac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:28:35.976284 31730 raft_consensus.cc:1261] T 17974244c9d94bd2b2dfcf4490f821b2 P bd5c10e848e34f599161b0a6f800ccde [term 1 FOLLOWER]: Refusing update from remote peer 8e53a55ef9bf491d9ac92ffd8dfb0fac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:28:35.976691 31821 consensus_queue.cc:1035] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [LEADER]: Connected to new peer: Peer: permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:28:35.976873 31818 consensus_queue.cc:1035] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:28:35.983785 31825 mvcc.cc:205] Tried to move back new op lower bound from 7316331380629315584 to 7316331380458356736. Current Snapshot: MvccSnapshot[applied={T|T < 7316331380629315584}]
I20260808 18:28:35.985024 31827 mvcc.cc:205] Tried to move back new op lower bound from 7316331380629315584 to 7316331380458356736. Current Snapshot: MvccSnapshot[applied={T|T < 7316331380629315584}]
I20260808 18:28:35.985889 31828 mvcc.cc:205] Tried to move back new op lower bound from 7316331380629315584 to 7316331380458356736. Current Snapshot: MvccSnapshot[applied={T|T < 7316331380629315584}]
I20260808 18:28:35.995682 31238 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:28:35.995937 31238 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:28:35.998603 31238 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b: Sending DeleteTablet for 3 replicas of tablet 17974244c9d94bd2b2dfcf4490f821b2
I20260808 18:28:35.999395 31390 tablet_service.cc:1514] Processing DeleteTablet for tablet 17974244c9d94bd2b2dfcf4490f821b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:35 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:54506
I20260808 18:28:35.999395 31551 tablet_service.cc:1514] Processing DeleteTablet for tablet 17974244c9d94bd2b2dfcf4490f821b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:35 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:40900
I20260808 18:28:35.999437 31710 tablet_service.cc:1514] Processing DeleteTablet for tablet 17974244c9d94bd2b2dfcf4490f821b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:35 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:38064
I20260808 18:28:35.999904 31836 tablet_replica.cc:330] T 17974244c9d94bd2b2dfcf4490f821b2 P 25886af7e7884fad86bbb0878c075c7c: stopping tablet replica
I20260808 18:28:36.000000 31837 tablet_replica.cc:330] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: stopping tablet replica
I20260808 18:28:36.000272 31836 raft_consensus.cc:2229] T 17974244c9d94bd2b2dfcf4490f821b2 P 25886af7e7884fad86bbb0878c075c7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:28:36.000272 31837 raft_consensus.cc:2229] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:28:36.000573 31836 raft_consensus.cc:2258] T 17974244c9d94bd2b2dfcf4490f821b2 P 25886af7e7884fad86bbb0878c075c7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:28:36.000587 31837 raft_consensus.cc:2258] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:28:36.000957 31238 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:28:36.001354 31238 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:28:36.002228 31836 ts_tablet_manager.cc:1905] T 17974244c9d94bd2b2dfcf4490f821b2 P 25886af7e7884fad86bbb0878c075c7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:28:36.002559 31837 ts_tablet_manager.cc:1905] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:28:36.005148 31837 ts_tablet_manager.cc:1918] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:28:36.005148 31836 ts_tablet_manager.cc:1918] T 17974244c9d94bd2b2dfcf4490f821b2 P 25886af7e7884fad86bbb0878c075c7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:28:36.005246 31836 log.cc:1196] T 17974244c9d94bd2b2dfcf4490f821b2 P 25886af7e7884fad86bbb0878c075c7c: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-0/wal/wals/17974244c9d94bd2b2dfcf4490f821b2
I20260808 18:28:36.005246 31837 log.cc:1196] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/17974244c9d94bd2b2dfcf4490f821b2
I20260808 18:28:36.005574 31837 ts_tablet_manager.cc:1939] T 17974244c9d94bd2b2dfcf4490f821b2 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: Deleting consensus metadata
I20260808 18:28:36.005574 31836 ts_tablet_manager.cc:1939] T 17974244c9d94bd2b2dfcf4490f821b2 P 25886af7e7884fad86bbb0878c075c7c: Deleting consensus metadata
I20260808 18:28:36.006661 31221 catalog_manager.cc:4841] TS 8e53a55ef9bf491d9ac92ffd8dfb0fac (127.29.78.130:39113): tablet 17974244c9d94bd2b2dfcf4490f821b2 (table test-table [id=c85b08913617490dabedffe30ef64c25]) successfully deleted
I20260808 18:28:36.006733 31390 tablet_service.cc:1467] Processing CreateTablet for tablet aa02e0867e2f46ca8afa89f899aad422 (DEFAULT_TABLE table=test-table [id=bd73765e294d4b5c85e3ae0e268fae4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:36.006837 31551 tablet_service.cc:1467] Processing CreateTablet for tablet aa02e0867e2f46ca8afa89f899aad422 (DEFAULT_TABLE table=test-table [id=bd73765e294d4b5c85e3ae0e268fae4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:36.006994 31390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa02e0867e2f46ca8afa89f899aad422. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:36.007071 31551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa02e0867e2f46ca8afa89f899aad422. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:36.007400 31710 tablet_service.cc:1467] Processing CreateTablet for tablet aa02e0867e2f46ca8afa89f899aad422 (DEFAULT_TABLE table=test-table [id=bd73765e294d4b5c85e3ae0e268fae4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:36.007665 31710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa02e0867e2f46ca8afa89f899aad422. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:36.009271 31812 tablet_bootstrap.cc:492] T aa02e0867e2f46ca8afa89f899aad422 P 25886af7e7884fad86bbb0878c075c7c: Bootstrap starting.
I20260808 18:28:36.010097 31812 tablet_bootstrap.cc:654] T aa02e0867e2f46ca8afa89f899aad422 P 25886af7e7884fad86bbb0878c075c7c: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:36.010126 31811 tablet_bootstrap.cc:492] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde: Bootstrap starting.
I20260808 18:28:36.010164 31813 tablet_bootstrap.cc:492] T aa02e0867e2f46ca8afa89f899aad422 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: Bootstrap starting.
I20260808 18:28:36.011081 31813 tablet_bootstrap.cc:654] T aa02e0867e2f46ca8afa89f899aad422 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:36.011114 31838 tablet_replica.cc:330] T 17974244c9d94bd2b2dfcf4490f821b2 P bd5c10e848e34f599161b0a6f800ccde: stopping tablet replica
I20260808 18:28:36.011198 31811 tablet_bootstrap.cc:654] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:36.011448 31838 raft_consensus.cc:2229] T 17974244c9d94bd2b2dfcf4490f821b2 P bd5c10e848e34f599161b0a6f800ccde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:28:36.011777 31838 raft_consensus.cc:2258] T 17974244c9d94bd2b2dfcf4490f821b2 P bd5c10e848e34f599161b0a6f800ccde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:28:36.012670 31811 tablet_bootstrap.cc:492] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde: No bootstrap required, opened a new log
I20260808 18:28:36.012768 31811 ts_tablet_manager.cc:1397] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260808 18:28:36.013192 31811 raft_consensus.cc:348] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } }
I20260808 18:28:36.013300 31811 raft_consensus.cc:374] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:36.013331 31811 raft_consensus.cc:729] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd5c10e848e34f599161b0a6f800ccde, State: Initialized, Role: FOLLOWER
I20260808 18:28:36.013339 31813 tablet_bootstrap.cc:492] T aa02e0867e2f46ca8afa89f899aad422 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: No bootstrap required, opened a new log
I20260808 18:28:36.013376 31838 ts_tablet_manager.cc:1905] T 17974244c9d94bd2b2dfcf4490f821b2 P bd5c10e848e34f599161b0a6f800ccde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:28:36.013408 31813 ts_tablet_manager.cc:1397] T aa02e0867e2f46ca8afa89f899aad422 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:28:36.013445 31811 consensus_queue.cc:260] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } }
I20260808 18:28:36.013705 31811 ts_tablet_manager.cc:1428] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 18:28:36.013813 31813 raft_consensus.cc:348] T aa02e0867e2f46ca8afa89f899aad422 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } }
I20260808 18:28:36.013931 31813 raft_consensus.cc:374] T aa02e0867e2f46ca8afa89f899aad422 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:36.013962 31813 raft_consensus.cc:729] T aa02e0867e2f46ca8afa89f899aad422 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e53a55ef9bf491d9ac92ffd8dfb0fac, State: Initialized, Role: FOLLOWER
I20260808 18:28:36.014067 31813 consensus_queue.cc:260] T aa02e0867e2f46ca8afa89f899aad422 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } }
I20260808 18:28:36.014285 31813 ts_tablet_manager.cc:1428] T aa02e0867e2f46ca8afa89f899aad422 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:28:36.014895 31223 catalog_manager.cc:4841] TS 25886af7e7884fad86bbb0878c075c7c (127.29.78.129:35115): tablet 17974244c9d94bd2b2dfcf4490f821b2 (table test-table [id=c85b08913617490dabedffe30ef64c25]) successfully deleted
I20260808 18:28:36.015725 31812 tablet_bootstrap.cc:492] T aa02e0867e2f46ca8afa89f899aad422 P 25886af7e7884fad86bbb0878c075c7c: No bootstrap required, opened a new log
I20260808 18:28:36.015818 31812 ts_tablet_manager.cc:1397] T aa02e0867e2f46ca8afa89f899aad422 P 25886af7e7884fad86bbb0878c075c7c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260808 18:28:36.016530 31812 raft_consensus.cc:348] T aa02e0867e2f46ca8afa89f899aad422 P 25886af7e7884fad86bbb0878c075c7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } }
I20260808 18:28:36.016633 31812 raft_consensus.cc:374] T aa02e0867e2f46ca8afa89f899aad422 P 25886af7e7884fad86bbb0878c075c7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:36.016675 31812 raft_consensus.cc:729] T aa02e0867e2f46ca8afa89f899aad422 P 25886af7e7884fad86bbb0878c075c7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25886af7e7884fad86bbb0878c075c7c, State: Initialized, Role: FOLLOWER
I20260808 18:28:36.016798 31812 consensus_queue.cc:260] T aa02e0867e2f46ca8afa89f899aad422 P 25886af7e7884fad86bbb0878c075c7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } }
I20260808 18:28:36.017086 31812 ts_tablet_manager.cc:1428] T aa02e0867e2f46ca8afa89f899aad422 P 25886af7e7884fad86bbb0878c075c7c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 18:28:36.017406 31838 ts_tablet_manager.cc:1918] T 17974244c9d94bd2b2dfcf4490f821b2 P bd5c10e848e34f599161b0a6f800ccde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:28:36.017524 31838 log.cc:1196] T 17974244c9d94bd2b2dfcf4490f821b2 P bd5c10e848e34f599161b0a6f800ccde: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/17974244c9d94bd2b2dfcf4490f821b2
I20260808 18:28:36.017890 31838 ts_tablet_manager.cc:1939] T 17974244c9d94bd2b2dfcf4490f821b2 P bd5c10e848e34f599161b0a6f800ccde: Deleting consensus metadata
I20260808 18:28:36.019040 31224 catalog_manager.cc:4841] TS bd5c10e848e34f599161b0a6f800ccde (127.29.78.131:40189): tablet 17974244c9d94bd2b2dfcf4490f821b2 (table test-table [id=c85b08913617490dabedffe30ef64c25]) successfully deleted
W20260808 18:28:36.034039 31786 tablet.cc:2369] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:28:36.072515 31817 raft_consensus.cc:482] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:28:36.072675 31817 raft_consensus.cc:504] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } }
W20260808 18:28:36.073773 31626 tablet.cc:2369] T aa02e0867e2f46ca8afa89f899aad422 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:28:36.073850 31817 leader_election.cc:288] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25886af7e7884fad86bbb0878c075c7c (127.29.78.129:35115), 8e53a55ef9bf491d9ac92ffd8dfb0fac (127.29.78.130:39113)
I20260808 18:28:36.078826 31665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:36.074033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:35115 (local address 127.29.78.131:33735)
0808 18:28:36.074323 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:36.074341 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:36.074460 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:36.074993 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:36.074996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:36.075018 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:36.075145 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:36.075151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.076219 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.076223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:36.077453 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:36.077463 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.078385 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.078388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:36.078524 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:36.078546 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 18:28:36.078613 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 18:28:36.078690 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":3}
I20260808 18:28:36.078826 31809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:36.074549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:58353 (local address 127.29.78.130:39113)
0808 18:28:36.074679 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:36.074684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:36.074805 (+ 121us) server_negotiation.cc:407] Connection header received
0808 18:28:36.074923 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:36.074926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:36.074984 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:36.075058 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:36.075376 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.076091 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.077414 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.078053 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.078192 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:36.078269 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:36.078645 (+ 376us) server_negotiation.cc:299] Negotiation successful
0808 18:28:36.078717 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 18:28:36.079236 31842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:36.074124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:39113 (local address 127.29.78.131:58353)
0808 18:28:36.074701 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:36.074715 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:36.074824 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:36.075091 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:36.075094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:36.075110 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:36.075256 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:36.075262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.076248 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.076251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:36.077283 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:36.077293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.078352 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.078359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:36.078488 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:36.078515 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 18:28:36.079026 (+ 511us) client_negotiation.cc:240] Negotiation successful
0808 18:28:36.079107 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":117,"threads_started":1}
I20260808 18:28:36.079325 31807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:36.074496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:33735 (local address 127.29.78.129:35115)
0808 18:28:36.074662 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:36.074665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:36.074678 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:28:36.074725 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:36.074727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:36.074786 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:36.074866 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:36.075271 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.076091 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.077583 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.078183 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.078287 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:36.078339 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:36.079105 (+ 766us) server_negotiation.cc:299] Negotiation successful
0808 18:28:36.079182 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 18:28:36.079699 31411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa02e0867e2f46ca8afa89f899aad422" candidate_uuid: "bd5c10e848e34f599161b0a6f800ccde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25886af7e7884fad86bbb0878c075c7c" is_pre_election: true
I20260808 18:28:36.079730 31559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa02e0867e2f46ca8afa89f899aad422" candidate_uuid: "bd5c10e848e34f599161b0a6f800ccde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" is_pre_election: true
I20260808 18:28:36.079865 31411 raft_consensus.cc:2454] T aa02e0867e2f46ca8afa89f899aad422 P 25886af7e7884fad86bbb0878c075c7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd5c10e848e34f599161b0a6f800ccde in term 0.
I20260808 18:28:36.079952 31559 raft_consensus.cc:2454] T aa02e0867e2f46ca8afa89f899aad422 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd5c10e848e34f599161b0a6f800ccde in term 0.
I20260808 18:28:36.080247 31659 leader_election.cc:302] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25886af7e7884fad86bbb0878c075c7c, bd5c10e848e34f599161b0a6f800ccde; no voters:
I20260808 18:28:36.080545 31817 raft_consensus.cc:2781] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:28:36.080613 31817 raft_consensus.cc:482] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:28:36.080652 31817 raft_consensus.cc:3037] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:36.081875 31817 raft_consensus.cc:504] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } }
I20260808 18:28:36.082237 31817 leader_election.cc:288] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde [CANDIDATE]: Term 1 election: Requested vote from peers 25886af7e7884fad86bbb0878c075c7c (127.29.78.129:35115), 8e53a55ef9bf491d9ac92ffd8dfb0fac (127.29.78.130:39113)
I20260808 18:28:36.082597 31411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa02e0867e2f46ca8afa89f899aad422" candidate_uuid: "bd5c10e848e34f599161b0a6f800ccde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25886af7e7884fad86bbb0878c075c7c"
I20260808 18:28:36.082605 31559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa02e0867e2f46ca8afa89f899aad422" candidate_uuid: "bd5c10e848e34f599161b0a6f800ccde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac"
I20260808 18:28:36.082715 31411 raft_consensus.cc:3037] T aa02e0867e2f46ca8afa89f899aad422 P 25886af7e7884fad86bbb0878c075c7c [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:36.082779 31559 raft_consensus.cc:3037] T aa02e0867e2f46ca8afa89f899aad422 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:36.083662 31559 raft_consensus.cc:2454] T aa02e0867e2f46ca8afa89f899aad422 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd5c10e848e34f599161b0a6f800ccde in term 1.
I20260808 18:28:36.083783 31411 raft_consensus.cc:2454] T aa02e0867e2f46ca8afa89f899aad422 P 25886af7e7884fad86bbb0878c075c7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd5c10e848e34f599161b0a6f800ccde in term 1.
I20260808 18:28:36.083949 31656 leader_election.cc:302] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e53a55ef9bf491d9ac92ffd8dfb0fac, bd5c10e848e34f599161b0a6f800ccde; no voters:
I20260808 18:28:36.084103 31817 raft_consensus.cc:2781] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:36.084329 31817 raft_consensus.cc:686] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde [term 1 LEADER]: Becoming Leader. State: Replica: bd5c10e848e34f599161b0a6f800ccde, State: Running, Role: LEADER
I20260808 18:28:36.084586 31817 consensus_queue.cc:237] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } }
I20260808 18:28:36.087128 31237 catalog_manager.cc:5495] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde reported cstate change: term changed from 0 to 1, leader changed from <none> to bd5c10e848e34f599161b0a6f800ccde (127.29.78.131). New cstate: current_term: 1 leader_uuid: "bd5c10e848e34f599161b0a6f800ccde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } health_report { overall_health: HEALTHY } } }
W20260808 18:28:36.107436 31465 tablet.cc:2369] T aa02e0867e2f46ca8afa89f899aad422 P 25886af7e7884fad86bbb0878c075c7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 18:28:36 dist-test-slave-gp6n krb5kdc[31188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213714, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:36.122066 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:36.112398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44466 (local address 127.29.78.190:33993)
0808 18:28:36.112555 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:36.112560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:36.112838 (+ 278us) server_negotiation.cc:407] Connection header received
0808 18:28:36.112992 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:36.112995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:36.113036 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:36.113106 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:36.113977 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.114553 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.115652 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.115854 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.115926 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:36.118604 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:36.118608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:36.118610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:36.118640 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:36.120087 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:36.120672 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:36.120674 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:36.120675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:36.120748 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:36.121054 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:36.121059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:36.121063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:36.121268 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:36.121344 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:36.121722 (+ 378us) server_negotiation.cc:299] Negotiation successful
0808 18:28:36.121923 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":64}
I20260808 18:28:36.122069 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:36.112332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:33993 (local address 127.0.0.1:44466)
0808 18:28:36.112726 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:36.112743 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:36.112884 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:36.113167 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:36.113172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:36.113728 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:36.113853 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:36.113861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.114665 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.114668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:36.115534 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:36.115546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.115976 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.115980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:36.116067 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:36.116656 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:36.116672 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:36.118459 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:36.120210 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:36.120220 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:36.120226 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:36.120568 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:36.120836 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:36.120839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:36.120841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:36.120948 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:36.121385 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:36.121392 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:36.121533 (+ 141us) client_negotiation.cc:769] Sending connection context
0808 18:28:36.121672 (+ 139us) client_negotiation.cc:240] Negotiation successful
0808 18:28:36.121844 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":131,"threads_started":1}
I20260808 18:28:36.126959 31237 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:44466:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 18:28:36.127812 31237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:28:36.132791 31551 tablet_service.cc:1467] Processing CreateTablet for tablet 6685d6c302f044e9abfbe5e841962935 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7d552163c4ce484e9731ef93a20ac02b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 18:28:36.132743 31390 tablet_service.cc:1467] Processing CreateTablet for tablet 6685d6c302f044e9abfbe5e841962935 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7d552163c4ce484e9731ef93a20ac02b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 18:28:36.132977 31710 tablet_service.cc:1467] Processing CreateTablet for tablet 6685d6c302f044e9abfbe5e841962935 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7d552163c4ce484e9731ef93a20ac02b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 18:28:36.133208 31551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6685d6c302f044e9abfbe5e841962935. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:36.133208 31390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6685d6c302f044e9abfbe5e841962935. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:36.133352 31710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6685d6c302f044e9abfbe5e841962935. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:36.136052 31812 tablet_bootstrap.cc:492] T 6685d6c302f044e9abfbe5e841962935 P 25886af7e7884fad86bbb0878c075c7c: Bootstrap starting.
I20260808 18:28:36.136801 31812 tablet_bootstrap.cc:654] T 6685d6c302f044e9abfbe5e841962935 P 25886af7e7884fad86bbb0878c075c7c: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:36.137336 31811 tablet_bootstrap.cc:492] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde: Bootstrap starting.
I20260808 18:28:36.138041 31813 tablet_bootstrap.cc:492] T 6685d6c302f044e9abfbe5e841962935 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: Bootstrap starting.
I20260808 18:28:36.138237 31811 tablet_bootstrap.cc:654] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:36.139036 31813 tablet_bootstrap.cc:654] T 6685d6c302f044e9abfbe5e841962935 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:36.139739 31812 tablet_bootstrap.cc:492] T 6685d6c302f044e9abfbe5e841962935 P 25886af7e7884fad86bbb0878c075c7c: No bootstrap required, opened a new log
I20260808 18:28:36.139842 31812 ts_tablet_manager.cc:1397] T 6685d6c302f044e9abfbe5e841962935 P 25886af7e7884fad86bbb0878c075c7c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260808 18:28:36.140345 31812 raft_consensus.cc:348] T 6685d6c302f044e9abfbe5e841962935 P 25886af7e7884fad86bbb0878c075c7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } }
I20260808 18:28:36.140462 31812 raft_consensus.cc:374] T 6685d6c302f044e9abfbe5e841962935 P 25886af7e7884fad86bbb0878c075c7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:36.140504 31812 raft_consensus.cc:729] T 6685d6c302f044e9abfbe5e841962935 P 25886af7e7884fad86bbb0878c075c7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25886af7e7884fad86bbb0878c075c7c, State: Initialized, Role: FOLLOWER
I20260808 18:28:36.140640 31812 consensus_queue.cc:260] T 6685d6c302f044e9abfbe5e841962935 P 25886af7e7884fad86bbb0878c075c7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } }
I20260808 18:28:36.140718 31813 tablet_bootstrap.cc:492] T 6685d6c302f044e9abfbe5e841962935 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: No bootstrap required, opened a new log
I20260808 18:28:36.140796 31813 ts_tablet_manager.cc:1397] T 6685d6c302f044e9abfbe5e841962935 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:28:36.140934 31812 ts_tablet_manager.cc:1428] T 6685d6c302f044e9abfbe5e841962935 P 25886af7e7884fad86bbb0878c075c7c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 18:28:36.141155 31819 tablet_replica.cc:439] T 6685d6c302f044e9abfbe5e841962935 P 25886af7e7884fad86bbb0878c075c7c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } } }
I20260808 18:28:36.141281 31819 tablet_replica.cc:442] T 6685d6c302f044e9abfbe5e841962935 P 25886af7e7884fad86bbb0878c075c7c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:28:36.141259 31813 raft_consensus.cc:348] T 6685d6c302f044e9abfbe5e841962935 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } }
I20260808 18:28:36.141372 31813 raft_consensus.cc:374] T 6685d6c302f044e9abfbe5e841962935 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:36.141404 31813 raft_consensus.cc:729] T 6685d6c302f044e9abfbe5e841962935 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e53a55ef9bf491d9ac92ffd8dfb0fac, State: Initialized, Role: FOLLOWER
I20260808 18:28:36.141748 31811 tablet_bootstrap.cc:492] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde: No bootstrap required, opened a new log
I20260808 18:28:36.141820 31811 ts_tablet_manager.cc:1397] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260808 18:28:36.141847 31813 consensus_queue.cc:260] T 6685d6c302f044e9abfbe5e841962935 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } }
I20260808 18:28:36.142107 31821 tablet_replica.cc:439] T 6685d6c302f044e9abfbe5e841962935 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } } }
I20260808 18:28:36.142236 31821 tablet_replica.cc:442] T 6685d6c302f044e9abfbe5e841962935 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:28:36.142304 31811 raft_consensus.cc:348] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } }
I20260808 18:28:36.142447 31811 raft_consensus.cc:374] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:36.142490 31811 raft_consensus.cc:729] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd5c10e848e34f599161b0a6f800ccde, State: Initialized, Role: FOLLOWER
I20260808 18:28:36.143060 31811 consensus_queue.cc:260] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } }
I20260808 18:28:36.143294 31817 tablet_replica.cc:439] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } } }
I20260808 18:28:36.143412 31817 tablet_replica.cc:442] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:28:36.143682 31811 ts_tablet_manager.cc:1428] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 18:28:36.144636 31813 ts_tablet_manager.cc:1428] T 6685d6c302f044e9abfbe5e841962935 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260808 18:28:36.184130 31817 raft_consensus.cc:482] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:28:36.184254 31817 raft_consensus.cc:504] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } }
I20260808 18:28:36.184675 31817 leader_election.cc:288] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25886af7e7884fad86bbb0878c075c7c (127.29.78.129:35115), 8e53a55ef9bf491d9ac92ffd8dfb0fac (127.29.78.130:39113)
I20260808 18:28:36.185097 31411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6685d6c302f044e9abfbe5e841962935" candidate_uuid: "bd5c10e848e34f599161b0a6f800ccde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25886af7e7884fad86bbb0878c075c7c" is_pre_election: true
I20260808 18:28:36.185101 31559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6685d6c302f044e9abfbe5e841962935" candidate_uuid: "bd5c10e848e34f599161b0a6f800ccde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" is_pre_election: true
I20260808 18:28:36.185232 31411 raft_consensus.cc:2454] T 6685d6c302f044e9abfbe5e841962935 P 25886af7e7884fad86bbb0878c075c7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd5c10e848e34f599161b0a6f800ccde in term 0.
I20260808 18:28:36.185266 31559 raft_consensus.cc:2454] T 6685d6c302f044e9abfbe5e841962935 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd5c10e848e34f599161b0a6f800ccde in term 0.
I20260808 18:28:36.185595 31659 leader_election.cc:302] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25886af7e7884fad86bbb0878c075c7c, bd5c10e848e34f599161b0a6f800ccde; no voters:
I20260808 18:28:36.185794 31817 raft_consensus.cc:2781] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:28:36.185854 31817 raft_consensus.cc:482] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:28:36.185883 31817 raft_consensus.cc:3037] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:36.187008 31817 raft_consensus.cc:504] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } }
I20260808 18:28:36.187330 31817 leader_election.cc:288] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde [CANDIDATE]: Term 1 election: Requested vote from peers 25886af7e7884fad86bbb0878c075c7c (127.29.78.129:35115), 8e53a55ef9bf491d9ac92ffd8dfb0fac (127.29.78.130:39113)
I20260808 18:28:36.187613 31411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6685d6c302f044e9abfbe5e841962935" candidate_uuid: "bd5c10e848e34f599161b0a6f800ccde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25886af7e7884fad86bbb0878c075c7c"
I20260808 18:28:36.187748 31411 raft_consensus.cc:3037] T 6685d6c302f044e9abfbe5e841962935 P 25886af7e7884fad86bbb0878c075c7c [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:36.187705 31559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6685d6c302f044e9abfbe5e841962935" candidate_uuid: "bd5c10e848e34f599161b0a6f800ccde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac"
I20260808 18:28:36.187830 31559 raft_consensus.cc:3037] T 6685d6c302f044e9abfbe5e841962935 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:36.188637 31411 raft_consensus.cc:2454] T 6685d6c302f044e9abfbe5e841962935 P 25886af7e7884fad86bbb0878c075c7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd5c10e848e34f599161b0a6f800ccde in term 1.
I20260808 18:28:36.188798 31559 raft_consensus.cc:2454] T 6685d6c302f044e9abfbe5e841962935 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd5c10e848e34f599161b0a6f800ccde in term 1.
I20260808 18:28:36.188901 31659 leader_election.cc:302] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25886af7e7884fad86bbb0878c075c7c, bd5c10e848e34f599161b0a6f800ccde; no voters:
I20260808 18:28:36.189107 31817 raft_consensus.cc:2781] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:36.189214 31817 raft_consensus.cc:686] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde [term 1 LEADER]: Becoming Leader. State: Replica: bd5c10e848e34f599161b0a6f800ccde, State: Running, Role: LEADER
I20260808 18:28:36.189369 31817 consensus_queue.cc:237] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } }
I20260808 18:28:36.190006 31843 tablet_replica.cc:439] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde: TxnStatusTablet state changed. Reason: New leader bd5c10e848e34f599161b0a6f800ccde. Latest consensus state: current_term: 1 leader_uuid: "bd5c10e848e34f599161b0a6f800ccde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } } }
I20260808 18:28:36.190130 31843 tablet_replica.cc:442] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde: This TxnStatusTablet replica's current role is: LEADER
I20260808 18:28:36.190649 31856 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 18:28:36.190869 31856 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 18:28:36.191088 31238 catalog_manager.cc:5495] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde reported cstate change: term changed from 0 to 1, leader changed from <none> to bd5c10e848e34f599161b0a6f800ccde (127.29.78.131). New cstate: current_term: 1 leader_uuid: "bd5c10e848e34f599161b0a6f800ccde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } health_report { overall_health: HEALTHY } } }
I20260808 18:28:36.213482 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:36.209547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38074 (local address 127.29.78.131:40189)
0808 18:28:36.209668 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:36.209672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:36.209689 (+ 17us) server_negotiation.cc:407] Connection header received
0808 18:28:36.209848 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:36.209852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:36.209915 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:36.210028 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:28:36.210298 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.211108 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.212042 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.212281 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.212393 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:36.212675 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:28:36.213086 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:28:36.213212 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:36.213284 (+ 72us) server_negotiation.cc:299] Negotiation successful
0808 18:28:36.213334 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 18:28:36.213472 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:36.209448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:40189 (local address 127.0.0.1:38074)
0808 18:28:36.209592 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:36.209610 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:36.209708 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:36.210035 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:36.210038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:36.210082 (+ 44us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 18:28:36.210189 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:36.210194 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.211228 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.211231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:36.211895 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:36.211904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.212392 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.212395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:36.212487 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:36.212529 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 18:28:36.213187 (+ 658us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 18:28:36.213203 (+ 16us) client_negotiation.cc:769] Sending connection context
0808 18:28:36.213273 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 18:28:36.213356 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260808 18:28:36.216432 31411 raft_consensus.cc:1261] T 6685d6c302f044e9abfbe5e841962935 P 25886af7e7884fad86bbb0878c075c7c [term 1 FOLLOWER]: Refusing update from remote peer bd5c10e848e34f599161b0a6f800ccde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:28:36.216665 31559 raft_consensus.cc:1261] T 6685d6c302f044e9abfbe5e841962935 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 1 FOLLOWER]: Refusing update from remote peer bd5c10e848e34f599161b0a6f800ccde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:28:36.217012 31843 consensus_queue.cc:1035] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde [LEADER]: Connected to new peer: Peer: permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:28:36.217406 31817 consensus_queue.cc:1035] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:28:36.221153 31821 tablet_replica.cc:439] T 6685d6c302f044e9abfbe5e841962935 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: TxnStatusTablet state changed. Reason: New leader bd5c10e848e34f599161b0a6f800ccde. Latest consensus state: current_term: 1 leader_uuid: "bd5c10e848e34f599161b0a6f800ccde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } } }
I20260808 18:28:36.221280 31821 tablet_replica.cc:442] T 6685d6c302f044e9abfbe5e841962935 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:28:36.222684 31843 tablet_replica.cc:439] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd5c10e848e34f599161b0a6f800ccde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } } }
I20260808 18:28:36.222786 31843 tablet_replica.cc:442] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde: This TxnStatusTablet replica's current role is: LEADER
I20260808 18:28:36.232254 31819 tablet_replica.cc:439] T 6685d6c302f044e9abfbe5e841962935 P 25886af7e7884fad86bbb0878c075c7c: TxnStatusTablet state changed. Reason: New leader bd5c10e848e34f599161b0a6f800ccde. Latest consensus state: current_term: 1 leader_uuid: "bd5c10e848e34f599161b0a6f800ccde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } } }
I20260808 18:28:36.232383 31819 tablet_replica.cc:442] T 6685d6c302f044e9abfbe5e841962935 P 25886af7e7884fad86bbb0878c075c7c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:28:36.232471 31819 tablet_replica.cc:439] T 6685d6c302f044e9abfbe5e841962935 P 25886af7e7884fad86bbb0878c075c7c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd5c10e848e34f599161b0a6f800ccde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } } }
I20260808 18:28:36.232542 31819 tablet_replica.cc:442] T 6685d6c302f044e9abfbe5e841962935 P 25886af7e7884fad86bbb0878c075c7c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:28:36.234546 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:36.229987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38084 (local address 127.29.78.131:40189)
0808 18:28:36.230103 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:36.230108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:36.230120 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:28:36.230157 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:36.230160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:36.230200 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:36.230259 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:28:36.230636 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.231409 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.232624 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.232930 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.233019 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:36.233812 (+ 793us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:28:36.233890 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:28:36.233951 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:36.234328 (+ 377us) server_negotiation.cc:299] Negotiation successful
0808 18:28:36.234380 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 18:28:36.235206 31817 tablet_replica.cc:439] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd5c10e848e34f599161b0a6f800ccde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } } }
I20260808 18:28:36.235316 31817 tablet_replica.cc:442] T 6685d6c302f044e9abfbe5e841962935 P bd5c10e848e34f599161b0a6f800ccde: This TxnStatusTablet replica's current role is: LEADER
I20260808 18:28:36.236224 31821 tablet_replica.cc:439] T 6685d6c302f044e9abfbe5e841962935 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd5c10e848e34f599161b0a6f800ccde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 } } peers { permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 } } peers { permanent_uuid: "bd5c10e848e34f599161b0a6f800ccde" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 40189 } } }
I20260808 18:28:36.236328 31821 tablet_replica.cc:442] T 6685d6c302f044e9abfbe5e841962935 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:28:36.245421 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:36.242327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:60223 (local address 127.29.78.131:40189)
0808 18:28:36.242450 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:36.242454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:36.242568 (+ 114us) server_negotiation.cc:407] Connection header received
0808 18:28:36.242708 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:36.242712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:36.242768 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:36.242829 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:28:36.243090 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.243563 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.244396 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.244601 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.244705 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:36.245000 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:28:36.245066 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:28:36.245140 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:36.245226 (+ 86us) server_negotiation.cc:299] Negotiation successful
0808 18:28:36.245278 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":41}
I20260808 18:28:36.245420 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:36.242286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:40189 (local address 127.29.78.131:60223)
0808 18:28:36.242459 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:36.242483 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:36.242599 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:36.242845 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:36.242848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:36.242853 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 18:28:36.242995 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:36.243001 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.243666 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.243672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:36.244276 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:36.244283 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.244701 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.244705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:36.244835 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:36.244871 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 18:28:36.245155 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 18:28:36.245161 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 18:28:36.245218 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 18:28:36.245308 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":56}
I20260808 18:28:36.253391 31411 raft_consensus.cc:1261] T aa02e0867e2f46ca8afa89f899aad422 P 25886af7e7884fad86bbb0878c075c7c [term 1 FOLLOWER]: Refusing update from remote peer bd5c10e848e34f599161b0a6f800ccde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:28:36.253546 31559 raft_consensus.cc:1261] T aa02e0867e2f46ca8afa89f899aad422 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 1 FOLLOWER]: Refusing update from remote peer bd5c10e848e34f599161b0a6f800ccde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:28:36.253882 31843 consensus_queue.cc:1035] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde [LEADER]: Connected to new peer: Peer: permanent_uuid: "25886af7e7884fad86bbb0878c075c7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:28:36.254133 31843 consensus_queue.cc:1035] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e53a55ef9bf491d9ac92ffd8dfb0fac" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:28:36.291052 31237 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:28:36.291230 31237 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:28:36.293268 31237 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 144cc6477b6d4c40ad5c304efe592b0b: Sending DeleteTablet for 3 replicas of tablet aa02e0867e2f46ca8afa89f899aad422
I20260808 18:28:36.293841 31390 tablet_service.cc:1514] Processing DeleteTablet for tablet aa02e0867e2f46ca8afa89f899aad422 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:36 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:54506
I20260808 18:28:36.293830 31551 tablet_service.cc:1514] Processing DeleteTablet for tablet aa02e0867e2f46ca8afa89f899aad422 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:36 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:40900
I20260808 18:28:36.293855 31709 tablet_service.cc:1514] Processing DeleteTablet for tablet aa02e0867e2f46ca8afa89f899aad422 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:36 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:38064
I20260808 18:28:36.294019 31838 tablet_replica.cc:330] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde: stopping tablet replica
I20260808 18:28:36.294020 31837 tablet_replica.cc:330] T aa02e0867e2f46ca8afa89f899aad422 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: stopping tablet replica
I20260808 18:28:36.294119 31837 raft_consensus.cc:2229] T aa02e0867e2f46ca8afa89f899aad422 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:28:36.294119 31838 raft_consensus.cc:2229] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:28:36.294205 31837 raft_consensus.cc:2258] T aa02e0867e2f46ca8afa89f899aad422 P 8e53a55ef9bf491d9ac92ffd8dfb0fac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:28:36.294376 31836 tablet_replica.cc:330] T aa02e0867e2f46ca8afa89f899aad422 P 25886af7e7884fad86bbb0878c075c7c: stopping tablet replica
I20260808 18:28:36.294443 31838 raft_consensus.cc:2258] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:28:36.294509 31836 raft_consensus.cc:2229] T aa02e0867e2f46ca8afa89f899aad422 P 25886af7e7884fad86bbb0878c075c7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:28:36.294634 31836 raft_consensus.cc:2258] T aa02e0867e2f46ca8afa89f899aad422 P 25886af7e7884fad86bbb0878c075c7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:28:36.295282 31838 ts_tablet_manager.cc:1905] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:28:36.295641 31837 ts_tablet_manager.cc:1905] T aa02e0867e2f46ca8afa89f899aad422 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:28:36.297421 31836 ts_tablet_manager.cc:1905] T aa02e0867e2f46ca8afa89f899aad422 P 25886af7e7884fad86bbb0878c075c7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:28:36.298368 31838 ts_tablet_manager.cc:1918] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:28:36.298471 31838 log.cc:1196] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/aa02e0867e2f46ca8afa89f899aad422
I20260808 18:28:36.298465 31837 ts_tablet_manager.cc:1918] T aa02e0867e2f46ca8afa89f899aad422 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:28:36.298528 31837 log.cc:1196] T aa02e0867e2f46ca8afa89f899aad422 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/aa02e0867e2f46ca8afa89f899aad422
I20260808 18:28:36.298754 31838 ts_tablet_manager.cc:1939] T aa02e0867e2f46ca8afa89f899aad422 P bd5c10e848e34f599161b0a6f800ccde: Deleting consensus metadata
I20260808 18:28:36.298779 31837 ts_tablet_manager.cc:1939] T aa02e0867e2f46ca8afa89f899aad422 P 8e53a55ef9bf491d9ac92ffd8dfb0fac: Deleting consensus metadata
I20260808 18:28:36.299569 31224 catalog_manager.cc:4841] TS bd5c10e848e34f599161b0a6f800ccde (127.29.78.131:40189): tablet aa02e0867e2f46ca8afa89f899aad422 (table test-table [id=bd73765e294d4b5c85e3ae0e268fae4c]) successfully deleted
I20260808 18:28:36.299897 31221 catalog_manager.cc:4841] TS 8e53a55ef9bf491d9ac92ffd8dfb0fac (127.29.78.130:39113): tablet aa02e0867e2f46ca8afa89f899aad422 (table test-table [id=bd73765e294d4b5c85e3ae0e268fae4c]) successfully deleted
I20260808 18:28:36.301512 31836 ts_tablet_manager.cc:1918] T aa02e0867e2f46ca8afa89f899aad422 P 25886af7e7884fad86bbb0878c075c7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:28:36.301594 31836 log.cc:1196] T aa02e0867e2f46ca8afa89f899aad422 P 25886af7e7884fad86bbb0878c075c7c: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786213708769829-30010-0/minicluster-data/ts-0/wal/wals/aa02e0867e2f46ca8afa89f899aad422
I20260808 18:28:36.301846 31836 ts_tablet_manager.cc:1939] T aa02e0867e2f46ca8afa89f899aad422 P 25886af7e7884fad86bbb0878c075c7c: Deleting consensus metadata
I20260808 18:28:36.302873 31223 catalog_manager.cc:4841] TS 25886af7e7884fad86bbb0878c075c7c (127.29.78.129:35115): tablet aa02e0867e2f46ca8afa89f899aad422 (table test-table [id=bd73765e294d4b5c85e3ae0e268fae4c]) successfully deleted
Aug 08 18:28:36 dist-test-slave-gp6n krb5kdc[31188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213715, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260808 18:28:36.313009 31807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:36.302373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.29.78.129:35115)
0808 18:28:36.302533 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:36.302538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:36.302774 (+ 236us) server_negotiation.cc:407] Connection header received
0808 18:28:36.302983 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:36.302986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:36.303041 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:36.303132 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:36.303834 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.304348 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.305208 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.305442 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.305558 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:36.309140 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:36.309149 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:36.309162 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:36.309186 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:36.311058 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:36.311573 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:36.311579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:36.311580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:36.311674 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:36.311992 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:36.311995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:36.311997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:36.312245 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:36.312475 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:36.312714 (+ 239us) server_negotiation.cc:299] Negotiation successful
0808 18:28:36.312840 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260808 18:28:36.313628 31457 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54512
I20260808 18:28:36.324121 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:36.315554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44476 (local address 127.29.78.190:33993)
0808 18:28:36.315682 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:36.315686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:36.315733 (+ 47us) server_negotiation.cc:407] Connection header received
0808 18:28:36.315867 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:36.315869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:36.315911 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:36.315980 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:36.316783 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.317310 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.318538 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:36.318919 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:36.319030 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:36.320566 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:36.320569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:36.320571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:36.320595 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:36.322310 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:36.322825 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:36.322828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:36.322829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:36.322907 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:36.323226 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:36.323229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:36.323230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:36.323432 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:36.323561 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:36.323817 (+ 256us) server_negotiation.cc:299] Negotiation successful
0808 18:28:36.323946 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260808 18:28:36.324666 31237 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44476
I20260808 18:28:36.326026 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 31318
I20260808 18:28:36.333877 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 31473
I20260808 18:28:36.345228 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 31636
I20260808 18:28:36.353920 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 31203
2026-08-08T18:28:36Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2796 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:28:36 dist-test-slave-gp6n krb5kdc[31877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:28:36 dist-test-slave-gp6n krb5kdc[31877](info): set up 2 sockets
Aug 08 18:28:36 dist-test-slave-gp6n krb5kdc[31877](info): commencing operation
krb5kdc: starting...
W20260808 18:28:37.529403 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:28:37 dist-test-slave-gp6n krb5kdc[31877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:28:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:28:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:28:37.654137 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:32829
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:32829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:37.763029 31891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:37.763322 31891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:37.763396 31891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:37.767350 31891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:28:37.767432 31891 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 18:28:37.767463 31891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:37.767513 31891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:37.767547 31891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:28:37.767598 31891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:28:37.772663 31891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:32829
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:32829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:37.773656 31891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:37.774677 31891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:37.782986 31900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:37.782996 31898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:37.782996 31897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:37.783423 31891 server_base.cc:1034] running on GCE node
I20260808 18:28:37.783996 31891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:37.785065 31891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:37.786223 31891 hybrid_clock.cc:648] HybridClock initialized: now 1786213717786193 us; error 40 us; skew 500 ppm
Aug 08 18:28:37 dist-test-slave-gp6n krb5kdc[31877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213717, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:37.788951 31891 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:28:37.789826 31891 webserver.cc:460] Webserver started at http://127.29.78.190:45043/ using document root <none> and password file <none>
I20260808 18:28:37.790356 31891 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:37.790475 31891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:37.790683 31891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:37.792671 31891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "7540f6de5b0c4525a2bcb20ae04ab9c0"
format_stamp: "Formatted at 2026-08-08 18:28:37 on dist-test-slave-gp6n"
server_key: "fdca15730e0cc5ac5e56866682a7f083"
server_key_iv: "be478c3d4a407a8d8852dfa7f6324ff7"
server_key_version: "encryptionkey@0"
I20260808 18:28:37.793299 31891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "7540f6de5b0c4525a2bcb20ae04ab9c0"
format_stamp: "Formatted at 2026-08-08 18:28:37 on dist-test-slave-gp6n"
server_key: "fdca15730e0cc5ac5e56866682a7f083"
server_key_iv: "be478c3d4a407a8d8852dfa7f6324ff7"
server_key_version: "encryptionkey@0"
I20260808 18:28:37.796949 31891 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260808 18:28:37.799288 31908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:37.800518 31891 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:28:37.800652 31891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "7540f6de5b0c4525a2bcb20ae04ab9c0"
format_stamp: "Formatted at 2026-08-08 18:28:37 on dist-test-slave-gp6n"
server_key: "fdca15730e0cc5ac5e56866682a7f083"
server_key_iv: "be478c3d4a407a8d8852dfa7f6324ff7"
server_key_version: "encryptionkey@0"
I20260808 18:28:37.800763 31891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:37.810648 31891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:37.813242 31891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:37.813439 31891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:37.821128 31891 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:32829
I20260808 18:28:37.821146 31977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:32829 every 8 connection(s)
I20260808 18:28:37.822155 31891 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:28:37.824872 31978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:37.830209 31978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0: Bootstrap starting.
I20260808 18:28:37.830673 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 31891
I20260808 18:28:37.830789 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:28:37.831054 30010 external_mini_cluster.cc:1428] Setting key d7e03f592426ef86747cac4ca88ddaa9
I20260808 18:28:37.832909 31978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:37.833710 31978 log.cc:824] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:37.835784 31978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0: No bootstrap required, opened a new log
Aug 08 18:28:37 dist-test-slave-gp6n krb5kdc[31877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:37.840436 31978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7540f6de5b0c4525a2bcb20ae04ab9c0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 32829 } }
I20260808 18:28:37.840739 31978 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:37.840819 31978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7540f6de5b0c4525a2bcb20ae04ab9c0, State: Initialized, Role: FOLLOWER
I20260808 18:28:37.841306 31978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7540f6de5b0c4525a2bcb20ae04ab9c0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 32829 } }
I20260808 18:28:37.841476 31978 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:28:37.841550 31978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:28:37.841665 31978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:37.842900 31978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7540f6de5b0c4525a2bcb20ae04ab9c0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 32829 } }
I20260808 18:28:37.843279 31978 leader_election.cc:302] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7540f6de5b0c4525a2bcb20ae04ab9c0; no voters:
I20260808 18:28:37.843657 31978 leader_election.cc:288] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:28:37.843724 31984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:37.843928 31984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0 [term 1 LEADER]: Becoming Leader. State: Replica: 7540f6de5b0c4525a2bcb20ae04ab9c0, State: Running, Role: LEADER
I20260808 18:28:37.844224 31984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7540f6de5b0c4525a2bcb20ae04ab9c0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 32829 } }
I20260808 18:28:37.844973 31978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:28:37.846026 31986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7540f6de5b0c4525a2bcb20ae04ab9c0. Latest consensus state: current_term: 1 leader_uuid: "7540f6de5b0c4525a2bcb20ae04ab9c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7540f6de5b0c4525a2bcb20ae04ab9c0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 32829 } } }
I20260808 18:28:37.846187 31986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0 [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:37.846660 31985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7540f6de5b0c4525a2bcb20ae04ab9c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7540f6de5b0c4525a2bcb20ae04ab9c0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 32829 } } }
I20260808 18:28:37.846771 31985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0 [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:37.847559 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:37.832369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56852 (local address 127.29.78.190:32829)
0808 18:28:37.832816 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:37.832838 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:37.832872 (+ 34us) server_negotiation.cc:407] Connection header received
0808 18:28:37.833546 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:37.833563 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:37.833935 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:37.834296 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:37.835136 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:37.836187 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:37.837404 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:37.837872 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:37.837992 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:37.841234 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:37.841240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:37.841256 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:37.841286 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:37.843367 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:37.844152 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:37.844158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:37.844165 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:37.844344 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:37.844706 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:37.844710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:37.844711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:37.845299 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:37.845571 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:37.846135 (+ 564us) server_negotiation.cc:299] Negotiation successful
0808 18:28:37.846487 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":149,"threads_started":1}
W20260808 18:28:37.850208 32000 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:28:37.850298 32000 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:28:37.850390 31994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:28:37.851155 31994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:28:37.857234 31994 catalog_manager.cc:1347] Generated new cluster ID: 63aa36ca86824b40b2700af5ded980bd
I20260808 18:28:37.857331 31994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:28:37.864293 31994 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:28:37.865204 31994 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:28:37.879536 31994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0: Generated new TSK 0
I20260808 18:28:37.880365 31994 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260808 18:28:37.917780 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:32829
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:38.035432 32006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:38.035686 32006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:38.035737 32006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:38.040150 32006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:38.040215 32006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:38.040257 32006 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 18:28:38.040315 32006 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 18:28:38.040382 32006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:28:38.046487 32006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.78.190:32829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:38.047686 32006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:38.048792 32006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:38.058684 32012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:38.058735 32018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:38.058708 32014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:38.058755 32006 server_base.cc:1034] running on GCE node
I20260808 18:28:38.059295 32006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:38.059940 32006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:38.061105 32006 hybrid_clock.cc:648] HybridClock initialized: now 1786213718061085 us; error 32 us; skew 500 ppm
Aug 08 18:28:38 dist-test-slave-gp6n krb5kdc[31877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213718, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:38.063982 32006 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:28:38.064862 32006 webserver.cc:460] Webserver started at http://127.29.78.129:45095/ using document root <none> and password file <none>
I20260808 18:28:38.065416 32006 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:38.065512 32006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:38.065744 32006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:38.067816 32006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "fa1a7cdb80ba44c48f80a60e6627f03b"
format_stamp: "Formatted at 2026-08-08 18:28:38 on dist-test-slave-gp6n"
server_key: "51bb82ad343e4bf11dd2b7f0bc828544"
server_key_iv: "29a5dead4cd3d072ac8f50db1e79195f"
server_key_version: "encryptionkey@0"
I20260808 18:28:38.068382 32006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "fa1a7cdb80ba44c48f80a60e6627f03b"
format_stamp: "Formatted at 2026-08-08 18:28:38 on dist-test-slave-gp6n"
server_key: "51bb82ad343e4bf11dd2b7f0bc828544"
server_key_iv: "29a5dead4cd3d072ac8f50db1e79195f"
server_key_version: "encryptionkey@0"
I20260808 18:28:38.072019 32006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 18:28:38.074318 32026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:38.075536 32006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:28:38.075678 32006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "fa1a7cdb80ba44c48f80a60e6627f03b"
format_stamp: "Formatted at 2026-08-08 18:28:38 on dist-test-slave-gp6n"
server_key: "51bb82ad343e4bf11dd2b7f0bc828544"
server_key_iv: "29a5dead4cd3d072ac8f50db1e79195f"
server_key_version: "encryptionkey@0"
I20260808 18:28:38.075790 32006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:38.092509 32006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:38.095840 32006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:38.096057 32006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:38.097465 32006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:38.097543 32006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:38.097611 32006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:38.097661 32006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 18:28:38 dist-test-slave-gp6n krb5kdc[31877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213718, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:38.109361 32006 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:34751
I20260808 18:28:38.109426 32157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:34751 every 8 connection(s)
I20260808 18:28:38.110240 32006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:28:38.111626 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.098503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:57793 (local address 127.29.78.190:32829)
0808 18:28:38.098676 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:38.098681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:38.099526 (+ 845us) server_negotiation.cc:407] Connection header received
0808 18:28:38.100854 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:38.100858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:38.100926 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:38.101042 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:38.102260 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.102950 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.104413 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.104848 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.104951 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.107917 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:38.107921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:38.107924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:38.107951 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:38.109440 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:38.109944 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:38.109949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:38.109951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:38.110043 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:38.110382 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:38.110387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:38.110389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:38.110615 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:38.110723 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:38.111304 (+ 581us) server_negotiation.cc:299] Negotiation successful
0808 18:28:38.111448 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 18:28:38.112349 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.098846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:32829 (local address 127.29.78.129:57793)
0808 18:28:38.099336 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:38.099398 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:38.100599 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:38.101225 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:38.101232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:38.101568 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:38.102020 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.102035 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.103089 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.103092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.104290 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.104299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.105000 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.105004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.105209 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.105859 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:38.105875 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:38.107757 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:38.109577 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:38.109582 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:38.109595 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:38.109838 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:38.110143 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:38.110146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:38.110148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:38.110284 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:38.110733 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:38.110742 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:38.111044 (+ 302us) client_negotiation.cc:769] Sending connection context
0808 18:28:38.111260 (+ 216us) client_negotiation.cc:240] Negotiation successful
0808 18:28:38.111563 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":109,"threads_started":1}
I20260808 18:28:38.113740 32158 heartbeater.cc:347] Connected to a master server at 127.29.78.190:32829
I20260808 18:28:38.113898 32158 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:38.114459 32158 heartbeater.cc:511] Master 127.29.78.190:32829 requested a full tablet report, sending...
I20260808 18:28:38.114964 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 32006
I20260808 18:28:38.115082 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:28:38.115381 30010 external_mini_cluster.cc:1428] Setting key 7b91a8871e1461db37f89dda96a8af6e
I20260808 18:28:38.116060 31928 ts_manager.cc:194] Registered new tserver with Master: fa1a7cdb80ba44c48f80a60e6627f03b (127.29.78.129:34751)
I20260808 18:28:38.117451 31928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:57793
I20260808 18:28:38.123911 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.115963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:32829 (local address 127.29.78.129:42687)
0808 18:28:38.116344 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:38.116358 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:38.116450 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:38.116683 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:38.116686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:38.116891 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:38.117015 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.117034 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.117997 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.118000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.118722 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.118731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.119366 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.119370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.119466 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.119986 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:38.120000 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:38.120363 (+ 363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:38.122354 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:38.122357 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:38.122359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:38.122678 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:38.122980 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:38.122982 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:38.122983 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:38.123061 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:38.123523 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:38.123525 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:38.123595 (+ 70us) client_negotiation.cc:769] Sending connection context
0808 18:28:38.123672 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 18:28:38.123772 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":86,"threads_started":1}
I20260808 18:28:38.124137 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.116039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:42687 (local address 127.29.78.190:32829)
0808 18:28:38.116162 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:38.116166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:38.116437 (+ 271us) server_negotiation.cc:407] Connection header received
0808 18:28:38.116544 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:38.116547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:38.116587 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:38.116662 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:38.117147 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.117845 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.118840 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.119237 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.119358 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.120505 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:38.120511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:38.120517 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:38.120548 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:38.122249 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:38.122769 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:38.122776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:38.122780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:38.122874 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:38.123153 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:38.123158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:38.123163 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:38.123419 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:38.123538 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:38.123766 (+ 228us) server_negotiation.cc:299] Negotiation successful
0808 18:28:38.123901 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260808 18:28:38.145251 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:32829
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:38.260802 32169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:38.261034 32169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:38.261078 32169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:38.264958 32169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:38.265009 32169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:38.265048 32169 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 18:28:38.265093 32169 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 18:28:38.265152 32169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:28:38.270181 32169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.78.190:32829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:38.271193 32169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:38.272133 32169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:38.281704 32177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:38.281678 32174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:38.281678 32175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:38.282112 32169 server_base.cc:1034] running on GCE node
I20260808 18:28:38.282509 32169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:38.283120 32169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:38.284299 32169 hybrid_clock.cc:648] HybridClock initialized: now 1786213718284272 us; error 46 us; skew 500 ppm
Aug 08 18:28:38 dist-test-slave-gp6n krb5kdc[31877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213718, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:38.287014 32169 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:28:38.287919 32169 webserver.cc:460] Webserver started at http://127.29.78.130:42379/ using document root <none> and password file <none>
I20260808 18:28:38.288494 32169 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:38.288589 32169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:38.288832 32169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:38.290977 32169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "d7b686b608e248cbb46aa83e75772a33"
format_stamp: "Formatted at 2026-08-08 18:28:38 on dist-test-slave-gp6n"
server_key: "71cb6b2cdf225861868d317979d00c63"
server_key_iv: "1455b3acbccd0b499da416c00451a10e"
server_key_version: "encryptionkey@0"
I20260808 18:28:38.291558 32169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "d7b686b608e248cbb46aa83e75772a33"
format_stamp: "Formatted at 2026-08-08 18:28:38 on dist-test-slave-gp6n"
server_key: "71cb6b2cdf225861868d317979d00c63"
server_key_iv: "1455b3acbccd0b499da416c00451a10e"
server_key_version: "encryptionkey@0"
I20260808 18:28:38.295289 32169 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:28:38.297662 32185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:38.298995 32169 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:28:38.299131 32169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "d7b686b608e248cbb46aa83e75772a33"
format_stamp: "Formatted at 2026-08-08 18:28:38 on dist-test-slave-gp6n"
server_key: "71cb6b2cdf225861868d317979d00c63"
server_key_iv: "1455b3acbccd0b499da416c00451a10e"
server_key_version: "encryptionkey@0"
I20260808 18:28:38.299247 32169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:38.315310 32169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:38.318121 32169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:38.318331 32169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:38.319756 32169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:38.319829 32169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:38.319895 32169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:38.319940 32169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 18:28:38 dist-test-slave-gp6n krb5kdc[31877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213718, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:38.331508 32169 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:44535
I20260808 18:28:38.331553 32316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:44535 every 8 connection(s)
I20260808 18:28:38.332607 32169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:28:38.334198 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.320630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:56191 (local address 127.29.78.190:32829)
0808 18:28:38.320762 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:38.320766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:38.321867 (+ 1101us) server_negotiation.cc:407] Connection header received
0808 18:28:38.323172 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:38.323179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:38.323248 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:38.323351 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:38.324631 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.325544 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.326935 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.327338 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.327476 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.330115 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:38.330121 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:38.330127 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:38.330162 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:38.331823 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:38.332363 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:38.332367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:38.332369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:38.332460 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:38.332791 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:38.332794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:38.332796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:38.333056 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:38.333237 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:38.333902 (+ 665us) server_negotiation.cc:299] Negotiation successful
0808 18:28:38.334046 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 18:28:38.335014 32197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.321173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:32829 (local address 127.29.78.130:56191)
0808 18:28:38.321697 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:38.321740 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:38.322837 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:38.323535 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:38.323542 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:38.324016 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:38.324493 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.324506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.325690 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.325694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.326810 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.326819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.327527 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.327534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.327717 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.328356 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:38.328374 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:38.329936 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:38.331955 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:38.331979 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:38.331994 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:38.332255 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:38.332555 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:38.332557 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:38.332558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:38.332685 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:38.333280 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:38.333287 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:38.333581 (+ 294us) client_negotiation.cc:769] Sending connection context
0808 18:28:38.333818 (+ 237us) client_negotiation.cc:240] Negotiation successful
0808 18:28:38.334178 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":144,"threads_started":1}
I20260808 18:28:38.336342 32317 heartbeater.cc:347] Connected to a master server at 127.29.78.190:32829
I20260808 18:28:38.336506 32317 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:38.337033 32317 heartbeater.cc:511] Master 127.29.78.190:32829 requested a full tablet report, sending...
I20260808 18:28:38.338093 31929 ts_manager.cc:194] Registered new tserver with Master: d7b686b608e248cbb46aa83e75772a33 (127.29.78.130:44535)
I20260808 18:28:38.338915 31929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:56191
I20260808 18:28:38.341581 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 32169
I20260808 18:28:38.341714 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:28:38.342011 30010 external_mini_cluster.cc:1428] Setting key 5be14106f508724baca71b5353fa2649
I20260808 18:28:38.345822 32323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.338307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:32829 (local address 127.29.78.130:40899)
0808 18:28:38.338662 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:38.338676 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:38.338770 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:38.338994 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:38.338998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:38.339200 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:38.339311 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.339317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.340088 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.340091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.340885 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.340893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.341287 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.341289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.341353 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.341845 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:38.341863 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:38.342224 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:38.343985 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:38.343991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:38.343996 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:38.344270 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:38.344568 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:38.344574 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:38.344578 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:38.344680 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:38.345103 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:38.345109 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:38.345249 (+ 140us) client_negotiation.cc:769] Sending connection context
0808 18:28:38.345368 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 18:28:38.345562 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":102,"threads_started":1}
I20260808 18:28:38.345841 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.338358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:40899 (local address 127.29.78.190:32829)
0808 18:28:38.338502 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:38.338506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:38.338751 (+ 245us) server_negotiation.cc:407] Connection header received
0808 18:28:38.338847 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:38.338849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:38.338885 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:38.338963 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:38.339399 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.339940 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.340991 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.341192 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.341269 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.342352 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:38.342357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:38.342360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:38.342386 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:38.343879 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:38.344365 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:38.344367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:38.344368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:38.344439 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:38.344769 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:38.344771 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:38.344773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:38.344974 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:38.345116 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:38.345430 (+ 314us) server_negotiation.cc:299] Negotiation successful
0808 18:28:38.345601 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260808 18:28:38.369470 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:32829
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:38.480940 32327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:38.481177 32327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:38.481222 32327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:38.485170 32327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:38.485224 32327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:38.485263 32327 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 18:28:38.485277 32327 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 18:28:38.485309 32327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:28:38.490379 32327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.78.190:32829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:38.491357 32327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:38.492326 32327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:38.502012 32333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:38.501963 32332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:38.502085 32335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:38.502126 32327 server_base.cc:1034] running on GCE node
I20260808 18:28:38.502641 32327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:38.503362 32327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:38.504544 32327 hybrid_clock.cc:648] HybridClock initialized: now 1786213718504526 us; error 48 us; skew 500 ppm
Aug 08 18:28:38 dist-test-slave-gp6n krb5kdc[31877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213718, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:38.507449 32327 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:28:38.508327 32327 webserver.cc:460] Webserver started at http://127.29.78.131:33521/ using document root <none> and password file <none>
I20260808 18:28:38.508872 32327 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:38.508961 32327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:38.509199 32327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:38.511298 32327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "2d70d72ef23146c7b87134cbd485e85a"
format_stamp: "Formatted at 2026-08-08 18:28:38 on dist-test-slave-gp6n"
server_key: "e0a4d420109ed85697ac136d7b736f9f"
server_key_iv: "829eb0fe934949d6fa34284bed844442"
server_key_version: "encryptionkey@0"
I20260808 18:28:38.511850 32327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "2d70d72ef23146c7b87134cbd485e85a"
format_stamp: "Formatted at 2026-08-08 18:28:38 on dist-test-slave-gp6n"
server_key: "e0a4d420109ed85697ac136d7b736f9f"
server_key_iv: "829eb0fe934949d6fa34284bed844442"
server_key_version: "encryptionkey@0"
I20260808 18:28:38.515604 32327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:28:38.517899 32342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:38.519055 32327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:28:38.519194 32327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "2d70d72ef23146c7b87134cbd485e85a"
format_stamp: "Formatted at 2026-08-08 18:28:38 on dist-test-slave-gp6n"
server_key: "e0a4d420109ed85697ac136d7b736f9f"
server_key_iv: "829eb0fe934949d6fa34284bed844442"
server_key_version: "encryptionkey@0"
I20260808 18:28:38.519307 32327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:38.545395 32327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:38.548517 32327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:38.548723 32327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:38.550201 32327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:38.550275 32327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:38.550348 32327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:38.550398 32327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 18:28:38 dist-test-slave-gp6n krb5kdc[31877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213718, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:38.561599 32327 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:43313
I20260808 18:28:38.561803 32472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:43313 every 8 connection(s)
I20260808 18:28:38.562765 32327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:28:38.563961 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.550960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:44093 (local address 127.29.78.190:32829)
0808 18:28:38.551143 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:38.551148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:38.552404 (+ 1256us) server_negotiation.cc:407] Connection header received
0808 18:28:38.553554 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:38.553557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:38.553625 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:38.553716 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:38.554936 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.555469 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.557046 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.557277 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.557397 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.559934 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:38.559937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:38.559939 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:38.559962 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:38.561666 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:38.562249 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:38.562252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:38.562254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:38.562331 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:38.562711 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:38.562714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:38.562715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:38.562936 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:38.563031 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:38.563667 (+ 636us) server_negotiation.cc:299] Negotiation successful
0808 18:28:38.563789 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 18:28:38.564706 32352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.551777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:32829 (local address 127.29.78.131:44093)
0808 18:28:38.552237 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:38.552288 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:38.553316 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:38.553894 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:38.553902 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:38.554244 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:38.554782 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.554796 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.555609 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.555612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.556913 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.556927 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.557452 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.557456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.557639 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.558255 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:38.558271 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:38.559758 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:38.561796 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:38.561801 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:38.561814 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:38.562134 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:38.562444 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:38.562447 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:38.562449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:38.562614 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:38.563051 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:38.563060 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:38.563364 (+ 304us) client_negotiation.cc:769] Sending connection context
0808 18:28:38.563602 (+ 238us) client_negotiation.cc:240] Negotiation successful
0808 18:28:38.563907 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":137,"threads_started":1}
I20260808 18:28:38.565972 32473 heartbeater.cc:347] Connected to a master server at 127.29.78.190:32829
I20260808 18:28:38.566169 32473 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:38.566322 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 32327
I20260808 18:28:38.566489 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:28:38.566869 30010 external_mini_cluster.cc:1428] Setting key ca8efe0a3ab4f27cbd863947515945b5
I20260808 18:28:38.566897 32473 heartbeater.cc:511] Master 127.29.78.190:32829 requested a full tablet report, sending...
I20260808 18:28:38.568125 31929 ts_manager.cc:194] Registered new tserver with Master: 2d70d72ef23146c7b87134cbd485e85a (127.29.78.131:43313)
I20260808 18:28:38.568881 31929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:44093
I20260808 18:28:38.569906 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:28:38.576413 32482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.568246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:32829 (local address 127.29.78.131:56659)
0808 18:28:38.568517 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:38.568533 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:38.568628 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:38.568833 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:38.568835 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:38.569036 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:38.569134 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.569139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.570211 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.570215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.571115 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.571129 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.571600 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.571606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.571738 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.572231 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:38.572251 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:38.572698 (+ 447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:38.574906 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:38.574909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:38.574910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:38.575096 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:38.575412 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:38.575414 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:38.575416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:38.575487 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:38.576012 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:38.576015 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:38.576098 (+ 83us) client_negotiation.cc:769] Sending connection context
0808 18:28:38.576180 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 18:28:38.576279 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":1,"thread_start_us":104,"threads_started":1}
I20260808 18:28:38.576609 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.568299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:56659 (local address 127.29.78.190:32829)
0808 18:28:38.568418 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:38.568422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:38.568615 (+ 193us) server_negotiation.cc:407] Connection header received
0808 18:28:38.568693 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:38.568696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:38.568736 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:38.568812 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:38.569261 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.570074 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.571251 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.571485 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.571572 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.572847 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:38.572853 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:38.572858 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:38.572892 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:38.574798 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:38.575198 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:38.575206 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:38.575211 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:38.575315 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:38.575597 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:38.575605 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:38.575610 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:38.575885 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:38.576021 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:38.576284 (+ 263us) server_negotiation.cc:299] Negotiation successful
0808 18:28:38.576418 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Aug 08 18:28:38 dist-test-slave-gp6n krb5kdc[31877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 18:28:38 dist-test-slave-gp6n krb5kdc[31877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:38.600873 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.592070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56856 (local address 127.29.78.190:32829)
0808 18:28:38.592214 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:38.592218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:38.592230 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:28:38.592266 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:38.592269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:38.592305 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:38.592401 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:38.593184 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.593743 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.594748 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.595005 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.595122 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.597651 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:38.597661 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:38.597663 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:38.597688 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:38.599059 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:38.599538 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:38.599548 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:38.599549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:38.599631 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:38.599960 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:38.599963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:38.599965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:38.600147 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:38.600301 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:38.600564 (+ 263us) server_negotiation.cc:299] Negotiation successful
0808 18:28:38.600690 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 18:28:38.603932 31928 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:28:38.607002 31928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:28:38.621800 32493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.615692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44535 (local address 127.0.0.1:53796)
0808 18:28:38.616231 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:38.616289 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:38.616412 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:38.617938 (+ 1526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:38.617940 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:38.617956 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:38.618072 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.618078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.619180 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.619184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.620325 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.620335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.621252 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.621256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.621405 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.621449 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 18:28:38.621539 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 18:28:38.621628 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":36,"thread_start_us":91,"threads_started":1}
I20260808 18:28:38.621855 32498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.616761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53796 (local address 127.29.78.130:44535)
0808 18:28:38.617506 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:38.617519 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:38.617539 (+ 20us) server_negotiation.cc:407] Connection header received
0808 18:28:38.617636 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:38.617640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:38.617823 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:38.617961 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:38.618186 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.619050 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.620443 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.621113 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.621367 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.621545 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:38.621601 (+ 56us) server_negotiation.cc:299] Negotiation successful
0808 18:28:38.621709 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":107,"threads_started":1}
I20260808 18:28:38.622469 32494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.615527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:43313 (local address 127.0.0.1:33218)
0808 18:28:38.616230 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:38.616288 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:38.616454 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:38.617234 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:38.617241 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:38.617262 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:38.617407 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.617415 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.618893 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.618896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.620746 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.620755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.621753 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.621756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.621856 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.622133 (+ 277us) client_negotiation.cc:769] Sending connection context
0808 18:28:38.622241 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 18:28:38.622313 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":47,"thread_start_us":60,"threads_started":1}
I20260808 18:28:38.622560 32495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.615679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33218 (local address 127.29.78.131:43313)
0808 18:28:38.616004 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:38.616021 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:38.616373 (+ 352us) server_negotiation.cc:407] Connection header received
0808 18:28:38.616615 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:38.616623 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:38.616838 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:38.616991 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:38.617531 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.618773 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.620867 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.621633 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.621738 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.621887 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:38.622286 (+ 399us) server_negotiation.cc:299] Negotiation successful
0808 18:28:38.622387 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":114,"threads_started":1}
I20260808 18:28:38.623510 32496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.616201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:34751 (local address 127.0.0.1:47270)
0808 18:28:38.617126 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:38.617142 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:38.617235 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:38.617645 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:38.617648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:38.617666 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:38.617754 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.617759 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.619212 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.619217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.620191 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.620200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.621222 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.621226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.622204 (+ 978us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.623126 (+ 922us) client_negotiation.cc:769] Sending connection context
0808 18:28:38.623210 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 18:28:38.623280 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"mutex_wait_us":24,"thread_start_us":85,"threads_started":1}
I20260808 18:28:38.623705 32497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.616732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47270 (local address 127.29.78.129:34751)
0808 18:28:38.617009 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:38.617014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:38.617229 (+ 215us) server_negotiation.cc:407] Connection header received
0808 18:28:38.617339 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:38.617342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:38.617500 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:38.617664 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:38.617852 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.619090 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.620312 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.621097 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.621223 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.621385 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:38.623262 (+ 1877us) server_negotiation.cc:299] Negotiation successful
0808 18:28:38.623364 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":95,"threads_started":1}
I20260808 18:28:38.625276 32245 tablet_service.cc:1467] Processing CreateTablet for tablet 04144c70223f4ef7a2c335e6466e36ba (DEFAULT_TABLE table=test-table [id=5823671c578645888012469a3e04d009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:38.625425 32398 tablet_service.cc:1467] Processing CreateTablet for tablet 04144c70223f4ef7a2c335e6466e36ba (DEFAULT_TABLE table=test-table [id=5823671c578645888012469a3e04d009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:38.626505 32245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04144c70223f4ef7a2c335e6466e36ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:38.626559 32398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04144c70223f4ef7a2c335e6466e36ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:38.626187 32083 tablet_service.cc:1467] Processing CreateTablet for tablet 04144c70223f4ef7a2c335e6466e36ba (DEFAULT_TABLE table=test-table [id=5823671c578645888012469a3e04d009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:38.627095 32083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04144c70223f4ef7a2c335e6466e36ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:38.633100 32499 tablet_bootstrap.cc:492] T 04144c70223f4ef7a2c335e6466e36ba P d7b686b608e248cbb46aa83e75772a33: Bootstrap starting.
I20260808 18:28:38.634455 32500 tablet_bootstrap.cc:492] T 04144c70223f4ef7a2c335e6466e36ba P fa1a7cdb80ba44c48f80a60e6627f03b: Bootstrap starting.
I20260808 18:28:38.634742 32501 tablet_bootstrap.cc:492] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a: Bootstrap starting.
I20260808 18:28:38.635763 32499 tablet_bootstrap.cc:654] T 04144c70223f4ef7a2c335e6466e36ba P d7b686b608e248cbb46aa83e75772a33: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:38.636354 32500 tablet_bootstrap.cc:654] T 04144c70223f4ef7a2c335e6466e36ba P fa1a7cdb80ba44c48f80a60e6627f03b: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:38.636626 32499 log.cc:824] T 04144c70223f4ef7a2c335e6466e36ba P d7b686b608e248cbb46aa83e75772a33: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:38.636682 32501 tablet_bootstrap.cc:654] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:38.637169 32500 log.cc:824] T 04144c70223f4ef7a2c335e6466e36ba P fa1a7cdb80ba44c48f80a60e6627f03b: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:38.637449 32501 log.cc:824] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:38.639235 32499 tablet_bootstrap.cc:492] T 04144c70223f4ef7a2c335e6466e36ba P d7b686b608e248cbb46aa83e75772a33: No bootstrap required, opened a new log
I20260808 18:28:38.639231 32500 tablet_bootstrap.cc:492] T 04144c70223f4ef7a2c335e6466e36ba P fa1a7cdb80ba44c48f80a60e6627f03b: No bootstrap required, opened a new log
I20260808 18:28:38.639438 32500 ts_tablet_manager.cc:1397] T 04144c70223f4ef7a2c335e6466e36ba P fa1a7cdb80ba44c48f80a60e6627f03b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260808 18:28:38.639430 32499 ts_tablet_manager.cc:1397] T 04144c70223f4ef7a2c335e6466e36ba P d7b686b608e248cbb46aa83e75772a33: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:28:38.639827 32501 tablet_bootstrap.cc:492] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a: No bootstrap required, opened a new log
I20260808 18:28:38.640044 32501 ts_tablet_manager.cc:1397] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 18:28:38.642803 32501 raft_consensus.cc:348] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } }
I20260808 18:28:38.643043 32501 raft_consensus.cc:374] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:38.643106 32501 raft_consensus.cc:729] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d70d72ef23146c7b87134cbd485e85a, State: Initialized, Role: FOLLOWER
I20260808 18:28:38.643374 32499 raft_consensus.cc:348] T 04144c70223f4ef7a2c335e6466e36ba P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } }
I20260808 18:28:38.643532 32501 consensus_queue.cc:260] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } }
I20260808 18:28:38.643644 32499 raft_consensus.cc:374] T 04144c70223f4ef7a2c335e6466e36ba P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:38.643501 32500 raft_consensus.cc:348] T 04144c70223f4ef7a2c335e6466e36ba P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } }
I20260808 18:28:38.643702 32499 raft_consensus.cc:729] T 04144c70223f4ef7a2c335e6466e36ba P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b686b608e248cbb46aa83e75772a33, State: Initialized, Role: FOLLOWER
I20260808 18:28:38.643711 32500 raft_consensus.cc:374] T 04144c70223f4ef7a2c335e6466e36ba P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:38.643759 32500 raft_consensus.cc:729] T 04144c70223f4ef7a2c335e6466e36ba P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1a7cdb80ba44c48f80a60e6627f03b, State: Initialized, Role: FOLLOWER
I20260808 18:28:38.644140 32499 consensus_queue.cc:260] T 04144c70223f4ef7a2c335e6466e36ba P d7b686b608e248cbb46aa83e75772a33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } }
I20260808 18:28:38.644140 32500 consensus_queue.cc:260] T 04144c70223f4ef7a2c335e6466e36ba P fa1a7cdb80ba44c48f80a60e6627f03b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } }
I20260808 18:28:38.644786 32158 heartbeater.cc:503] Master 127.29.78.190:32829 was elected leader, sending a full tablet report...
I20260808 18:28:38.645076 32501 ts_tablet_manager.cc:1428] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 18:28:38.645102 32499 ts_tablet_manager.cc:1428] T 04144c70223f4ef7a2c335e6466e36ba P d7b686b608e248cbb46aa83e75772a33: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 18:28:38.645146 32500 ts_tablet_manager.cc:1428] T 04144c70223f4ef7a2c335e6466e36ba P fa1a7cdb80ba44c48f80a60e6627f03b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:28:38.645370 32473 heartbeater.cc:503] Master 127.29.78.190:32829 was elected leader, sending a full tablet report...
I20260808 18:28:38.646324 32317 heartbeater.cc:503] Master 127.29.78.190:32829 was elected leader, sending a full tablet report...
W20260808 18:28:38.813786 32474 tablet.cc:2369] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:28:38.818095 32505 raft_consensus.cc:482] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:28:38.818322 32505 raft_consensus.cc:504] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } }
I20260808 18:28:38.819571 32505 leader_election.cc:288] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7b686b608e248cbb46aa83e75772a33 (127.29.78.130:44535), fa1a7cdb80ba44c48f80a60e6627f03b (127.29.78.129:34751)
I20260808 18:28:38.824186 32352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.819868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44535 (local address 127.29.78.131:43465)
0808 18:28:38.820000 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:38.820025 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:38.820132 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:38.820434 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:38.820438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:38.820458 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:38.820588 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.820594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.821531 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.821536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.822799 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.822808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.823582 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.823586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.823851 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.823877 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 18:28:38.823960 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 18:28:38.824049 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":43}
I20260808 18:28:38.824186 32498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.819940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:43465 (local address 127.29.78.130:44535)
0808 18:28:38.820086 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:38.820090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:38.820124 (+ 34us) server_negotiation.cc:407] Connection header received
0808 18:28:38.820256 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:38.820260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:38.820308 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:38.820418 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:38.820702 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.821391 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.822918 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.823480 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.823597 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.823653 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:38.824001 (+ 348us) server_negotiation.cc:299] Negotiation successful
0808 18:28:38.824078 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 18:28:38.825089 32267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04144c70223f4ef7a2c335e6466e36ba" candidate_uuid: "2d70d72ef23146c7b87134cbd485e85a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b686b608e248cbb46aa83e75772a33" is_pre_election: true
I20260808 18:28:38.825343 32508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.820171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:34751 (local address 127.29.78.131:37189)
0808 18:28:38.820635 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:38.820646 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:38.820763 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:38.821083 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:38.821086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:38.821105 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:38.821693 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.821703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.822745 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.822748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.823966 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.823976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.824966 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.824969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.825068 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.825087 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 18:28:38.825152 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 18:28:38.825213 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":168,"threads_started":1}
I20260808 18:28:38.825389 32497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.820365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:37189 (local address 127.29.78.129:34751)
0808 18:28:38.820501 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:38.820505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:38.820729 (+ 224us) server_negotiation.cc:407] Connection header received
0808 18:28:38.820902 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:38.820906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:38.820956 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:38.821047 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:38.821831 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.822635 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.824105 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.824724 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.824812 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.824856 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:38.825196 (+ 340us) server_negotiation.cc:299] Negotiation successful
0808 18:28:38.825277 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 18:28:38.825417 32267 raft_consensus.cc:2454] T 04144c70223f4ef7a2c335e6466e36ba P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d70d72ef23146c7b87134cbd485e85a in term 0.
I20260808 18:28:38.826011 32344 leader_election.cc:302] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d70d72ef23146c7b87134cbd485e85a, d7b686b608e248cbb46aa83e75772a33; no voters:
I20260808 18:28:38.826159 32109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04144c70223f4ef7a2c335e6466e36ba" candidate_uuid: "2d70d72ef23146c7b87134cbd485e85a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" is_pre_election: true
I20260808 18:28:38.826321 32505 raft_consensus.cc:2781] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:28:38.826478 32109 raft_consensus.cc:2454] T 04144c70223f4ef7a2c335e6466e36ba P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d70d72ef23146c7b87134cbd485e85a in term 0.
I20260808 18:28:38.826535 32505 raft_consensus.cc:482] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:28:38.826589 32505 raft_consensus.cc:3037] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:38.827880 32505 raft_consensus.cc:504] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } }
I20260808 18:28:38.828243 32505 leader_election.cc:288] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a [CANDIDATE]: Term 1 election: Requested vote from peers d7b686b608e248cbb46aa83e75772a33 (127.29.78.130:44535), fa1a7cdb80ba44c48f80a60e6627f03b (127.29.78.129:34751)
I20260808 18:28:38.828531 32267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04144c70223f4ef7a2c335e6466e36ba" candidate_uuid: "2d70d72ef23146c7b87134cbd485e85a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b686b608e248cbb46aa83e75772a33"
I20260808 18:28:38.828646 32109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04144c70223f4ef7a2c335e6466e36ba" candidate_uuid: "2d70d72ef23146c7b87134cbd485e85a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b"
I20260808 18:28:38.828701 32267 raft_consensus.cc:3037] T 04144c70223f4ef7a2c335e6466e36ba P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:38.828745 32109 raft_consensus.cc:3037] T 04144c70223f4ef7a2c335e6466e36ba P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:38.830025 32109 raft_consensus.cc:2454] T 04144c70223f4ef7a2c335e6466e36ba P fa1a7cdb80ba44c48f80a60e6627f03b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d70d72ef23146c7b87134cbd485e85a in term 1.
I20260808 18:28:38.830032 32267 raft_consensus.cc:2454] T 04144c70223f4ef7a2c335e6466e36ba P d7b686b608e248cbb46aa83e75772a33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d70d72ef23146c7b87134cbd485e85a in term 1.
I20260808 18:28:38.830319 32344 leader_election.cc:302] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d70d72ef23146c7b87134cbd485e85a, fa1a7cdb80ba44c48f80a60e6627f03b; no voters:
I20260808 18:28:38.830545 32505 raft_consensus.cc:2781] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:38.830765 32505 raft_consensus.cc:686] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a [term 1 LEADER]: Becoming Leader. State: Replica: 2d70d72ef23146c7b87134cbd485e85a, State: Running, Role: LEADER
I20260808 18:28:38.831077 32505 consensus_queue.cc:237] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } }
I20260808 18:28:38.834373 31929 catalog_manager.cc:5495] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d70d72ef23146c7b87134cbd485e85a (127.29.78.131). New cstate: current_term: 1 leader_uuid: "2d70d72ef23146c7b87134cbd485e85a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } health_report { overall_health: UNKNOWN } } }
W20260808 18:28:38.834811 32318 tablet.cc:2369] T 04144c70223f4ef7a2c335e6466e36ba P d7b686b608e248cbb46aa83e75772a33: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 18:28:38.862124 32160 tablet.cc:2369] T 04144c70223f4ef7a2c335e6466e36ba P fa1a7cdb80ba44c48f80a60e6627f03b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 18:28:38 dist-test-slave-gp6n krb5kdc[31877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213717, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:38.876283 32518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.868195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:32829 (local address 127.0.0.1:56872)
0808 18:28:38.868507 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:38.868520 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:38.868667 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:38.868909 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:38.868911 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:38.869352 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:38.869446 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.869452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.870198 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.870203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.870822 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:38.870830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.871257 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.871260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:38.871347 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.871922 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:38.871934 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:38.873378 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:38.874923 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:38.874931 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:38.874936 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:38.875166 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:38.875405 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:38.875407 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:38.875408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:38.875485 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:38.875809 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:38.875814 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:38.875911 (+ 97us) client_negotiation.cc:769] Sending connection context
0808 18:28:38.875997 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 18:28:38.876104 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":114,"threads_started":1}
I20260808 18:28:38.876282 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:38.868241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.29.78.190:32829)
0808 18:28:38.868384 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:38.868391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:38.868603 (+ 212us) server_negotiation.cc:407] Connection header received
0808 18:28:38.868763 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:38.868768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:38.868807 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:38.868893 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:38.869568 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.870088 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.870946 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:38.871150 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:38.871257 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:38.873514 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:38.873517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:38.873520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:38.873542 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:38.874825 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:38.875251 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:38.875254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:38.875255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:38.875327 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:38.875560 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:38.875562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:38.875564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:38.875723 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:38.875796 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:38.876053 (+ 257us) server_negotiation.cc:299] Negotiation successful
0808 18:28:38.876157 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":44}
I20260808 18:28:38.880124 31929 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:56872:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 18:28:38.880831 31929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:28:38.885413 32083 tablet_service.cc:1467] Processing CreateTablet for tablet 7058cc277750430bad415039f04f4d72 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1bd7c0b8f33d42d3b9e865f760074cb9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 18:28:38.885536 32398 tablet_service.cc:1467] Processing CreateTablet for tablet 7058cc277750430bad415039f04f4d72 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1bd7c0b8f33d42d3b9e865f760074cb9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 18:28:38.885514 32245 tablet_service.cc:1467] Processing CreateTablet for tablet 7058cc277750430bad415039f04f4d72 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1bd7c0b8f33d42d3b9e865f760074cb9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 18:28:38.886178 32083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7058cc277750430bad415039f04f4d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:38.886206 32398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7058cc277750430bad415039f04f4d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:38.886206 32245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7058cc277750430bad415039f04f4d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:38.888624 32500 tablet_bootstrap.cc:492] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b: Bootstrap starting.
I20260808 18:28:38.889066 32501 tablet_bootstrap.cc:492] T 7058cc277750430bad415039f04f4d72 P 2d70d72ef23146c7b87134cbd485e85a: Bootstrap starting.
I20260808 18:28:38.889060 32499 tablet_bootstrap.cc:492] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33: Bootstrap starting.
I20260808 18:28:38.889717 32500 tablet_bootstrap.cc:654] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:38.890031 32499 tablet_bootstrap.cc:654] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:38.890038 32501 tablet_bootstrap.cc:654] T 7058cc277750430bad415039f04f4d72 P 2d70d72ef23146c7b87134cbd485e85a: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:38.891307 32499 tablet_bootstrap.cc:492] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33: No bootstrap required, opened a new log
I20260808 18:28:38.891417 32499 ts_tablet_manager.cc:1397] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 18:28:38.891458 32500 tablet_bootstrap.cc:492] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b: No bootstrap required, opened a new log
I20260808 18:28:38.891487 32501 tablet_bootstrap.cc:492] T 7058cc277750430bad415039f04f4d72 P 2d70d72ef23146c7b87134cbd485e85a: No bootstrap required, opened a new log
I20260808 18:28:38.891557 32500 ts_tablet_manager.cc:1397] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:28:38.891557 32501 ts_tablet_manager.cc:1397] T 7058cc277750430bad415039f04f4d72 P 2d70d72ef23146c7b87134cbd485e85a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:28:38.892004 32501 raft_consensus.cc:348] T 7058cc277750430bad415039f04f4d72 P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:38.892014 32500 raft_consensus.cc:348] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:38.892125 32500 raft_consensus.cc:374] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:38.892125 32501 raft_consensus.cc:374] T 7058cc277750430bad415039f04f4d72 P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:38.892066 32499 raft_consensus.cc:348] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:38.892161 32500 raft_consensus.cc:729] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1a7cdb80ba44c48f80a60e6627f03b, State: Initialized, Role: FOLLOWER
I20260808 18:28:38.892161 32501 raft_consensus.cc:729] T 7058cc277750430bad415039f04f4d72 P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d70d72ef23146c7b87134cbd485e85a, State: Initialized, Role: FOLLOWER
I20260808 18:28:38.892186 32499 raft_consensus.cc:374] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:38.892218 32499 raft_consensus.cc:729] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b686b608e248cbb46aa83e75772a33, State: Initialized, Role: FOLLOWER
I20260808 18:28:38.892287 32501 consensus_queue.cc:260] T 7058cc277750430bad415039f04f4d72 P 2d70d72ef23146c7b87134cbd485e85a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:38.892287 32500 consensus_queue.cc:260] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:38.892331 32499 consensus_queue.cc:260] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:38.892550 32500 ts_tablet_manager.cc:1428] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:28:38.892560 32501 ts_tablet_manager.cc:1428] T 7058cc277750430bad415039f04f4d72 P 2d70d72ef23146c7b87134cbd485e85a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:28:38.892624 32499 ts_tablet_manager.cc:1428] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:28:38.892607 32507 tablet_replica.cc:439] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } } }
I20260808 18:28:38.892741 32507 tablet_replica.cc:442] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:28:38.892750 32505 tablet_replica.cc:439] T 7058cc277750430bad415039f04f4d72 P 2d70d72ef23146c7b87134cbd485e85a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } } }
I20260808 18:28:38.892872 32505 tablet_replica.cc:442] T 7058cc277750430bad415039f04f4d72 P 2d70d72ef23146c7b87134cbd485e85a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:28:38.892807 32506 tablet_replica.cc:439] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } } }
I20260808 18:28:38.892925 32506 tablet_replica.cc:442] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:28:39.089411 32507 raft_consensus.cc:482] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:28:39.089555 32507 raft_consensus.cc:504] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:39.090794 32507 leader_election.cc:288] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa1a7cdb80ba44c48f80a60e6627f03b (127.29.78.129:34751), 2d70d72ef23146c7b87134cbd485e85a (127.29.78.131:43313)
I20260808 18:28:39.094641 32506 raft_consensus.cc:482] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:28:39.094775 32506 raft_consensus.cc:504] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:39.096060 32506 leader_election.cc:288] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7b686b608e248cbb46aa83e75772a33 (127.29.78.130:44535), 2d70d72ef23146c7b87134cbd485e85a (127.29.78.131:43313)
I20260808 18:28:39.096168 32495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:39.091145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:57303 (local address 127.29.78.131:43313)
0808 18:28:39.091278 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:39.091284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:39.091508 (+ 224us) server_negotiation.cc:407] Connection header received
0808 18:28:39.091648 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:39.091652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:39.091707 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:39.091802 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:39.092145 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.093033 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.095043 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.095614 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.095754 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:39.095809 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:39.095972 (+ 163us) server_negotiation.cc:299] Negotiation successful
0808 18:28:39.096057 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 18:28:39.096194 32522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:39.091084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:43313 (local address 127.29.78.130:57303)
0808 18:28:39.091392 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:39.091406 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:39.091531 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:39.091823 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:39.091826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:39.091852 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:39.092013 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:39.092027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.093159 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.093162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:39.094922 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:39.094934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.095751 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.095754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:39.095831 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:39.095851 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 18:28:39.095935 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 18:28:39.096015 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":333,"spinlock_wait_cycles":63104,"thread_start_us":75,"threads_started":1}
I20260808 18:28:39.096724 32411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7058cc277750430bad415039f04f4d72" candidate_uuid: "d7b686b608e248cbb46aa83e75772a33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d70d72ef23146c7b87134cbd485e85a" is_pre_election: true
I20260808 18:28:39.096966 32411 raft_consensus.cc:2454] T 7058cc277750430bad415039f04f4d72 P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7b686b608e248cbb46aa83e75772a33 in term 0.
I20260808 18:28:39.097079 32497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:39.091282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:37885 (local address 127.29.78.129:34751)
0808 18:28:39.091488 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:39.091495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:39.091513 (+ 18us) server_negotiation.cc:407] Connection header received
0808 18:28:39.091648 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:39.091652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:39.091707 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:39.091802 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:39.092145 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.093033 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.095870 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.096491 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.096618 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:39.096662 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:39.096882 (+ 220us) server_negotiation.cc:299] Negotiation successful
0808 18:28:39.096960 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260808 18:28:39.097492 32188 leader_election.cc:302] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d70d72ef23146c7b87134cbd485e85a, d7b686b608e248cbb46aa83e75772a33; no voters:
I20260808 18:28:39.097826 32507 raft_consensus.cc:2781] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:28:39.097860 32523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:39.091116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:34751 (local address 127.29.78.130:37885)
0808 18:28:39.091392 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:39.091406 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:39.091535 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:39.091823 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:39.091826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:39.091852 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:39.092020 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:39.092027 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.093159 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.093162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:39.094922 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:39.094934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.096612 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.096616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:39.096712 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:39.096737 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 18:28:39.097599 (+ 862us) client_negotiation.cc:240] Negotiation successful
0808 18:28:39.097691 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":190,"spinlock_wait_cycles":13440,"thread_start_us":86,"threads_started":1}
I20260808 18:28:39.097883 32507 raft_consensus.cc:482] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:28:39.097934 32507 raft_consensus.cc:3037] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:39.099301 32507 raft_consensus.cc:504] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:39.099596 32507 leader_election.cc:288] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33 [CANDIDATE]: Term 1 election: Requested vote from peers fa1a7cdb80ba44c48f80a60e6627f03b (127.29.78.129:34751), 2d70d72ef23146c7b87134cbd485e85a (127.29.78.131:43313)
I20260808 18:28:39.099923 32411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7058cc277750430bad415039f04f4d72" candidate_uuid: "d7b686b608e248cbb46aa83e75772a33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d70d72ef23146c7b87134cbd485e85a"
I20260808 18:28:39.100052 32411 raft_consensus.cc:3037] T 7058cc277750430bad415039f04f4d72 P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:39.100411 32107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7058cc277750430bad415039f04f4d72" candidate_uuid: "d7b686b608e248cbb46aa83e75772a33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b"
I20260808 18:28:39.100522 32107 raft_consensus.cc:3037] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:39.101316 32411 raft_consensus.cc:2454] T 7058cc277750430bad415039f04f4d72 P 2d70d72ef23146c7b87134cbd485e85a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b686b608e248cbb46aa83e75772a33 in term 1.
I20260808 18:28:39.101415 32498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:39.096833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:44073 (local address 127.29.78.130:44535)
0808 18:28:39.096932 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:39.096936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:39.097516 (+ 580us) server_negotiation.cc:407] Connection header received
0808 18:28:39.097699 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:39.097702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:39.097741 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:39.097784 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:39.098153 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.098683 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.100187 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.100743 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.100824 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:39.100885 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:39.101248 (+ 363us) server_negotiation.cc:299] Negotiation successful
0808 18:28:39.101313 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 18:28:39.101548 32109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7058cc277750430bad415039f04f4d72" candidate_uuid: "d7b686b608e248cbb46aa83e75772a33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" is_pre_election: true
I20260808 18:28:39.101706 32107 raft_consensus.cc:2454] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b686b608e248cbb46aa83e75772a33 in term 1.
I20260808 18:28:39.101847 32526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:39.096456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44535 (local address 127.29.78.129:44073)
0808 18:28:39.097433 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:39.097447 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:39.097587 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:39.097907 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:39.097910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:39.097931 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:39.098049 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:39.098054 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.098789 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.098793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:39.100053 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:39.100063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.100994 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.100999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:39.101115 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:39.101137 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 18:28:39.101528 (+ 391us) client_negotiation.cc:240] Negotiation successful
0808 18:28:39.101614 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"thread_start_us":116,"threads_started":1}
I20260808 18:28:39.102180 32525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:39.097138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:43313 (local address 127.29.78.129:50825)
0808 18:28:39.097557 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:39.097572 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:39.097709 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:39.098082 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:39.098085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:39.098104 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:39.098683 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:39.098693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.099688 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.099691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:39.100801 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:39.100810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.101746 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.101750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:39.101848 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:39.101876 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 18:28:39.101993 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 18:28:39.102059 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":99,"threads_started":1}
I20260808 18:28:39.102197 32495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:39.097386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:50825 (local address 127.29.78.131:43313)
0808 18:28:39.097572 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:39.097576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:39.097643 (+ 67us) server_negotiation.cc:407] Connection header received
0808 18:28:39.097830 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:39.097835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:39.097886 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:39.097939 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:39.098835 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.099558 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.100934 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.101512 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.101612 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:39.101653 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:39.102035 (+ 382us) server_negotiation.cc:299] Negotiation successful
0808 18:28:39.102100 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260808 18:28:39.102149 32187 leader_election.cc:302] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7b686b608e248cbb46aa83e75772a33, fa1a7cdb80ba44c48f80a60e6627f03b; no voters:
I20260808 18:28:39.102357 32507 raft_consensus.cc:2781] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:39.102574 32507 raft_consensus.cc:686] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33 [term 1 LEADER]: Becoming Leader. State: Replica: d7b686b608e248cbb46aa83e75772a33, State: Running, Role: LEADER
I20260808 18:28:39.102592 32267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7058cc277750430bad415039f04f4d72" candidate_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b686b608e248cbb46aa83e75772a33" is_pre_election: true
I20260808 18:28:39.102943 32507 consensus_queue.cc:237] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:39.103626 32411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7058cc277750430bad415039f04f4d72" candidate_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d70d72ef23146c7b87134cbd485e85a" is_pre_election: true
I20260808 18:28:39.103804 32411 raft_consensus.cc:2379] T 7058cc277750430bad415039f04f4d72 P 2d70d72ef23146c7b87134cbd485e85a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa1a7cdb80ba44c48f80a60e6627f03b in current term 1: Already voted for candidate d7b686b608e248cbb46aa83e75772a33 in this term.
I20260808 18:28:39.104292 32029 leader_election.cc:302] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa1a7cdb80ba44c48f80a60e6627f03b; no voters: 2d70d72ef23146c7b87134cbd485e85a, d7b686b608e248cbb46aa83e75772a33
I20260808 18:28:39.104548 32506 raft_consensus.cc:2726] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 18:28:39.105876 32527 tablet_replica.cc:439] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33: TxnStatusTablet state changed. Reason: New leader d7b686b608e248cbb46aa83e75772a33. Latest consensus state: current_term: 1 leader_uuid: "d7b686b608e248cbb46aa83e75772a33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } } }
I20260808 18:28:39.106576 31927 catalog_manager.cc:5495] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7b686b608e248cbb46aa83e75772a33 (127.29.78.130). New cstate: current_term: 1 leader_uuid: "d7b686b608e248cbb46aa83e75772a33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } health_report { overall_health: UNKNOWN } } }
I20260808 18:28:39.107196 32527 tablet_replica.cc:442] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33: This TxnStatusTablet replica's current role is: LEADER
I20260808 18:28:39.107519 32530 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 18:28:39.107708 32530 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 18:28:39.132409 32518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:39.128362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44535 (local address 127.0.0.1:53808)
0808 18:28:39.128529 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:39.128547 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:39.128655 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:39.128992 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:39.128994 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:39.128999 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 18:28:39.129088 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:39.129092 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.130086 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.130088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:39.130770 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:39.130779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.131352 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.131355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:39.131417 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:39.131460 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 18:28:39.132166 (+ 706us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 18:28:39.132178 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 18:28:39.132228 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 18:28:39.132295 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260808 18:28:39.132431 32498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:39.128458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53808 (local address 127.29.78.130:44535)
0808 18:28:39.128595 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:39.128599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:39.128615 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:28:39.128803 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:39.128806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:39.128869 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:39.128973 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:28:39.129182 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.129978 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.130879 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.131240 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.131337 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:39.131602 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:28:39.132076 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:28:39.132185 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:39.132243 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 18:28:39.132296 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 18:28:39.137668 32411 raft_consensus.cc:1261] T 7058cc277750430bad415039f04f4d72 P 2d70d72ef23146c7b87134cbd485e85a [term 1 FOLLOWER]: Refusing update from remote peer d7b686b608e248cbb46aa83e75772a33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:28:39.137661 32109 raft_consensus.cc:1261] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b [term 1 FOLLOWER]: Refusing update from remote peer d7b686b608e248cbb46aa83e75772a33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:28:39.138222 32507 consensus_queue.cc:1035] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:28:39.138373 32527 consensus_queue.cc:1035] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:28:39.141830 32505 tablet_replica.cc:439] T 7058cc277750430bad415039f04f4d72 P 2d70d72ef23146c7b87134cbd485e85a: TxnStatusTablet state changed. Reason: New leader d7b686b608e248cbb46aa83e75772a33. Latest consensus state: current_term: 1 leader_uuid: "d7b686b608e248cbb46aa83e75772a33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } } }
I20260808 18:28:39.141973 32505 tablet_replica.cc:442] T 7058cc277750430bad415039f04f4d72 P 2d70d72ef23146c7b87134cbd485e85a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:28:39.142285 32506 tablet_replica.cc:439] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b: TxnStatusTablet state changed. Reason: New leader d7b686b608e248cbb46aa83e75772a33. Latest consensus state: current_term: 1 leader_uuid: "d7b686b608e248cbb46aa83e75772a33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } } }
I20260808 18:28:39.142391 32506 tablet_replica.cc:442] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:28:39.144317 32527 tablet_replica.cc:439] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7b686b608e248cbb46aa83e75772a33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } } }
I20260808 18:28:39.144423 32527 tablet_replica.cc:442] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33: This TxnStatusTablet replica's current role is: LEADER
I20260808 18:28:39.145751 32506 tablet_replica.cc:439] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7b686b608e248cbb46aa83e75772a33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } } }
I20260808 18:28:39.145885 32506 tablet_replica.cc:442] T 7058cc277750430bad415039f04f4d72 P fa1a7cdb80ba44c48f80a60e6627f03b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:28:39.146076 32538 mvcc.cc:205] Tried to move back new op lower bound from 7316331393580904448 to 7316331393448263680. Current Snapshot: MvccSnapshot[applied={T|T < 7316331393580904448}]
I20260808 18:28:39.146484 32527 tablet_replica.cc:439] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7b686b608e248cbb46aa83e75772a33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } } }
I20260808 18:28:39.146592 32527 tablet_replica.cc:442] T 7058cc277750430bad415039f04f4d72 P d7b686b608e248cbb46aa83e75772a33: This TxnStatusTablet replica's current role is: LEADER
I20260808 18:28:39.147508 32505 tablet_replica.cc:439] T 7058cc277750430bad415039f04f4d72 P 2d70d72ef23146c7b87134cbd485e85a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7b686b608e248cbb46aa83e75772a33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } } }
I20260808 18:28:39.147634 32505 tablet_replica.cc:442] T 7058cc277750430bad415039f04f4d72 P 2d70d72ef23146c7b87134cbd485e85a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:28:39.157374 32495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:39.153414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33226 (local address 127.29.78.131:43313)
0808 18:28:39.153535 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:39.153538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:39.153750 (+ 212us) server_negotiation.cc:407] Connection header received
0808 18:28:39.153879 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:39.153881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:39.153927 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:39.154013 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:28:39.154256 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.154857 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.156077 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.156280 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.156350 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:39.156645 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:28:39.156999 (+ 354us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:28:39.157084 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:39.157188 (+ 104us) server_negotiation.cc:299] Negotiation successful
0808 18:28:39.157232 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:28:39.168114 32498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:39.164457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:49785 (local address 127.29.78.130:44535)
0808 18:28:39.164591 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:39.164596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:39.164855 (+ 259us) server_negotiation.cc:407] Connection header received
0808 18:28:39.165018 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:39.165025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:39.165067 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:39.165148 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:28:39.165392 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.165975 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.167008 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.167232 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.167340 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:39.167692 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:28:39.167758 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:28:39.167837 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:39.167929 (+ 92us) server_negotiation.cc:299] Negotiation successful
0808 18:28:39.167979 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 18:28:39.168138 32552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:39.164365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44535 (local address 127.29.78.131:49785)
0808 18:28:39.164761 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:39.164773 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:39.164892 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:39.165158 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:39.165161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:39.165169 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 18:28:39.165276 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:39.165288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.166084 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.166089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:39.166885 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:39.166894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.167380 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.167386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:39.167478 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:39.167519 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 18:28:39.167845 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 18:28:39.167854 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 18:28:39.167927 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 18:28:39.168002 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":121,"threads_started":1}
I20260808 18:28:39.174966 32109 raft_consensus.cc:1261] T 04144c70223f4ef7a2c335e6466e36ba P fa1a7cdb80ba44c48f80a60e6627f03b [term 1 FOLLOWER]: Refusing update from remote peer 2d70d72ef23146c7b87134cbd485e85a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:28:39.175185 32267 raft_consensus.cc:1261] T 04144c70223f4ef7a2c335e6466e36ba P d7b686b608e248cbb46aa83e75772a33 [term 1 FOLLOWER]: Refusing update from remote peer 2d70d72ef23146c7b87134cbd485e85a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:28:39.175439 32505 consensus_queue.cc:1035] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:28:39.176023 32505 consensus_queue.cc:1035] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:28:39.179646 32536 mvcc.cc:205] Tried to move back new op lower bound from 7316331393736269824 to 7316331392334639104. Current Snapshot: MvccSnapshot[applied={T|T < 7316331393736269824}]
I20260808 18:28:39.185950 32535 mvcc.cc:205] Tried to move back new op lower bound from 7316331393736269824 to 7316331392334639104. Current Snapshot: MvccSnapshot[applied={T|T < 7316331393736269824}]
I20260808 18:28:39.198257 32495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:39.194713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:57933 (local address 127.29.78.131:43313)
0808 18:28:39.194820 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:39.194824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:39.195116 (+ 292us) server_negotiation.cc:407] Connection header received
0808 18:28:39.195267 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:39.195270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:39.195320 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:39.195408 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:28:39.195635 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.196151 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.197090 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.197323 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.197431 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:39.197728 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:28:39.197806 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:28:39.197888 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:39.198070 (+ 182us) server_negotiation.cc:299] Negotiation successful
0808 18:28:39.198119 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 18:28:39.198272 32557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:39.194635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:43313 (local address 127.29.78.130:57933)
0808 18:28:39.195013 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:39.195030 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:39.195164 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:39.195412 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:39.195414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:39.195421 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 18:28:39.195525 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:39.195529 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.196260 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.196265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:39.196952 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:39.196962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.197432 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.197436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:39.197538 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:39.197578 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 18:28:39.197931 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 18:28:39.197940 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 18:28:39.198037 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 18:28:39.198118 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":103,"threads_started":1}
I20260808 18:28:39.223357 31928 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:28:39.223583 31928 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:28:39.226470 31928 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0: Sending DeleteTablet for 3 replicas of tablet 04144c70223f4ef7a2c335e6466e36ba
I20260808 18:28:39.227130 32398 tablet_service.cc:1514] Processing DeleteTablet for tablet 04144c70223f4ef7a2c335e6466e36ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:39 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:33218
I20260808 18:28:39.227416 32083 tablet_service.cc:1514] Processing DeleteTablet for tablet 04144c70223f4ef7a2c335e6466e36ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:39 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:47270
I20260808 18:28:39.227941 32561 tablet_replica.cc:330] T 04144c70223f4ef7a2c335e6466e36ba P fa1a7cdb80ba44c48f80a60e6627f03b: stopping tablet replica
I20260808 18:28:39.228319 32561 raft_consensus.cc:2229] T 04144c70223f4ef7a2c335e6466e36ba P fa1a7cdb80ba44c48f80a60e6627f03b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:28:39.228641 32561 raft_consensus.cc:2258] T 04144c70223f4ef7a2c335e6466e36ba P fa1a7cdb80ba44c48f80a60e6627f03b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:28:39.228646 31928 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:28:39.229077 31928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:28:39.230046 32245 tablet_service.cc:1514] Processing DeleteTablet for tablet 04144c70223f4ef7a2c335e6466e36ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:39 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:53796
I20260808 18:28:39.230543 32561 ts_tablet_manager.cc:1905] T 04144c70223f4ef7a2c335e6466e36ba P fa1a7cdb80ba44c48f80a60e6627f03b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:28:39.233660 32561 ts_tablet_manager.cc:1918] T 04144c70223f4ef7a2c335e6466e36ba P fa1a7cdb80ba44c48f80a60e6627f03b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:28:39.233820 32561 log.cc:1196] T 04144c70223f4ef7a2c335e6466e36ba P fa1a7cdb80ba44c48f80a60e6627f03b: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-0/wal/wals/04144c70223f4ef7a2c335e6466e36ba
I20260808 18:28:39.234220 32561 ts_tablet_manager.cc:1939] T 04144c70223f4ef7a2c335e6466e36ba P fa1a7cdb80ba44c48f80a60e6627f03b: Deleting consensus metadata
I20260808 18:28:39.234719 32398 tablet_service.cc:1467] Processing CreateTablet for tablet 0cc9eb66eb6e4122a932da1c85843889 (DEFAULT_TABLE table=test-table [id=403768197c15454d8873945894912445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:39.234999 32398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cc9eb66eb6e4122a932da1c85843889. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:39.237453 32501 tablet_bootstrap.cc:492] T 0cc9eb66eb6e4122a932da1c85843889 P 2d70d72ef23146c7b87134cbd485e85a: Bootstrap starting.
I20260808 18:28:39.238070 31910 catalog_manager.cc:4841] TS fa1a7cdb80ba44c48f80a60e6627f03b (127.29.78.129:34751): tablet 04144c70223f4ef7a2c335e6466e36ba (table test-table [id=5823671c578645888012469a3e04d009]) successfully deleted
I20260808 18:28:39.238441 32083 tablet_service.cc:1467] Processing CreateTablet for tablet 0cc9eb66eb6e4122a932da1c85843889 (DEFAULT_TABLE table=test-table [id=403768197c15454d8873945894912445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:39.238373 32501 tablet_bootstrap.cc:654] T 0cc9eb66eb6e4122a932da1c85843889 P 2d70d72ef23146c7b87134cbd485e85a: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:39.238693 32083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cc9eb66eb6e4122a932da1c85843889. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:39.239746 32501 tablet_bootstrap.cc:492] T 0cc9eb66eb6e4122a932da1c85843889 P 2d70d72ef23146c7b87134cbd485e85a: No bootstrap required, opened a new log
I20260808 18:28:39.239825 32501 ts_tablet_manager.cc:1397] T 0cc9eb66eb6e4122a932da1c85843889 P 2d70d72ef23146c7b87134cbd485e85a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 18:28:39.240223 32501 raft_consensus.cc:348] T 0cc9eb66eb6e4122a932da1c85843889 P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:39.240321 32501 raft_consensus.cc:374] T 0cc9eb66eb6e4122a932da1c85843889 P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:39.240351 32501 raft_consensus.cc:729] T 0cc9eb66eb6e4122a932da1c85843889 P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d70d72ef23146c7b87134cbd485e85a, State: Initialized, Role: FOLLOWER
I20260808 18:28:39.240461 32501 consensus_queue.cc:260] T 0cc9eb66eb6e4122a932da1c85843889 P 2d70d72ef23146c7b87134cbd485e85a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:39.240722 32501 ts_tablet_manager.cc:1428] T 0cc9eb66eb6e4122a932da1c85843889 P 2d70d72ef23146c7b87134cbd485e85a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 18:28:39.241655 32245 tablet_service.cc:1467] Processing CreateTablet for tablet 0cc9eb66eb6e4122a932da1c85843889 (DEFAULT_TABLE table=test-table [id=403768197c15454d8873945894912445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:39.241952 32245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cc9eb66eb6e4122a932da1c85843889. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:39.242583 32560 tablet_replica.cc:330] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a: stopping tablet replica
I20260808 18:28:39.242887 32560 raft_consensus.cc:2229] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:28:39.243233 32560 raft_consensus.cc:2258] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:28:39.244539 32560 ts_tablet_manager.cc:1905] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:28:39.246039 32500 tablet_bootstrap.cc:492] T 0cc9eb66eb6e4122a932da1c85843889 P fa1a7cdb80ba44c48f80a60e6627f03b: Bootstrap starting.
I20260808 18:28:39.247080 32500 tablet_bootstrap.cc:654] T 0cc9eb66eb6e4122a932da1c85843889 P fa1a7cdb80ba44c48f80a60e6627f03b: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:39.247787 32560 ts_tablet_manager.cc:1918] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:28:39.247934 32560 log.cc:1196] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/04144c70223f4ef7a2c335e6466e36ba
I20260808 18:28:39.248131 32499 tablet_bootstrap.cc:492] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33: Bootstrap starting.
I20260808 18:28:39.248353 32560 ts_tablet_manager.cc:1939] T 04144c70223f4ef7a2c335e6466e36ba P 2d70d72ef23146c7b87134cbd485e85a: Deleting consensus metadata
I20260808 18:28:39.249133 32499 tablet_bootstrap.cc:654] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:39.250497 31911 catalog_manager.cc:4841] TS 2d70d72ef23146c7b87134cbd485e85a (127.29.78.131:43313): tablet 04144c70223f4ef7a2c335e6466e36ba (table test-table [id=5823671c578645888012469a3e04d009]) successfully deleted
I20260808 18:28:39.251171 32500 tablet_bootstrap.cc:492] T 0cc9eb66eb6e4122a932da1c85843889 P fa1a7cdb80ba44c48f80a60e6627f03b: No bootstrap required, opened a new log
I20260808 18:28:39.251223 32499 tablet_bootstrap.cc:492] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33: No bootstrap required, opened a new log
I20260808 18:28:39.251255 32500 ts_tablet_manager.cc:1397] T 0cc9eb66eb6e4122a932da1c85843889 P fa1a7cdb80ba44c48f80a60e6627f03b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260808 18:28:39.251310 32499 ts_tablet_manager.cc:1397] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:28:39.251694 32500 raft_consensus.cc:348] T 0cc9eb66eb6e4122a932da1c85843889 P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:39.251799 32500 raft_consensus.cc:374] T 0cc9eb66eb6e4122a932da1c85843889 P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:39.251834 32500 raft_consensus.cc:729] T 0cc9eb66eb6e4122a932da1c85843889 P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1a7cdb80ba44c48f80a60e6627f03b, State: Initialized, Role: FOLLOWER
I20260808 18:28:39.251783 32499 raft_consensus.cc:348] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:39.251900 32499 raft_consensus.cc:374] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:39.251936 32499 raft_consensus.cc:729] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b686b608e248cbb46aa83e75772a33, State: Initialized, Role: FOLLOWER
I20260808 18:28:39.251952 32500 consensus_queue.cc:260] T 0cc9eb66eb6e4122a932da1c85843889 P fa1a7cdb80ba44c48f80a60e6627f03b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:39.252454 32499 consensus_queue.cc:260] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:39.254168 32500 ts_tablet_manager.cc:1428] T 0cc9eb66eb6e4122a932da1c85843889 P fa1a7cdb80ba44c48f80a60e6627f03b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260808 18:28:39.254643 32499 ts_tablet_manager.cc:1428] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:28:39.257390 32562 tablet_replica.cc:330] T 04144c70223f4ef7a2c335e6466e36ba P d7b686b608e248cbb46aa83e75772a33: stopping tablet replica
I20260808 18:28:39.258734 32562 raft_consensus.cc:2229] T 04144c70223f4ef7a2c335e6466e36ba P d7b686b608e248cbb46aa83e75772a33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:28:39.258934 32562 raft_consensus.cc:2258] T 04144c70223f4ef7a2c335e6466e36ba P d7b686b608e248cbb46aa83e75772a33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:28:39.260658 32562 ts_tablet_manager.cc:1905] T 04144c70223f4ef7a2c335e6466e36ba P d7b686b608e248cbb46aa83e75772a33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:28:39.264775 32562 ts_tablet_manager.cc:1918] T 04144c70223f4ef7a2c335e6466e36ba P d7b686b608e248cbb46aa83e75772a33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:28:39.264891 32562 log.cc:1196] T 04144c70223f4ef7a2c335e6466e36ba P d7b686b608e248cbb46aa83e75772a33: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/04144c70223f4ef7a2c335e6466e36ba
I20260808 18:28:39.265280 32562 ts_tablet_manager.cc:1939] T 04144c70223f4ef7a2c335e6466e36ba P d7b686b608e248cbb46aa83e75772a33: Deleting consensus metadata
I20260808 18:28:39.266719 31910 catalog_manager.cc:4841] TS d7b686b608e248cbb46aa83e75772a33 (127.29.78.130:44535): tablet 04144c70223f4ef7a2c335e6466e36ba (table test-table [id=5823671c578645888012469a3e04d009]) successfully deleted
I20260808 18:28:39.395624 32527 raft_consensus.cc:482] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:28:39.395767 32527 raft_consensus.cc:504] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:39.396257 32527 leader_election.cc:288] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa1a7cdb80ba44c48f80a60e6627f03b (127.29.78.129:34751), 2d70d72ef23146c7b87134cbd485e85a (127.29.78.131:43313)
I20260808 18:28:39.396733 32411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cc9eb66eb6e4122a932da1c85843889" candidate_uuid: "d7b686b608e248cbb46aa83e75772a33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d70d72ef23146c7b87134cbd485e85a" is_pre_election: true
I20260808 18:28:39.396766 32109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cc9eb66eb6e4122a932da1c85843889" candidate_uuid: "d7b686b608e248cbb46aa83e75772a33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" is_pre_election: true
I20260808 18:28:39.396895 32411 raft_consensus.cc:2454] T 0cc9eb66eb6e4122a932da1c85843889 P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7b686b608e248cbb46aa83e75772a33 in term 0.
I20260808 18:28:39.396934 32109 raft_consensus.cc:2454] T 0cc9eb66eb6e4122a932da1c85843889 P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7b686b608e248cbb46aa83e75772a33 in term 0.
I20260808 18:28:39.397230 32188 leader_election.cc:302] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d70d72ef23146c7b87134cbd485e85a, d7b686b608e248cbb46aa83e75772a33; no voters:
I20260808 18:28:39.397405 32527 raft_consensus.cc:2781] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:28:39.397464 32527 raft_consensus.cc:482] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:28:39.397523 32527 raft_consensus.cc:3037] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:39.398599 32527 raft_consensus.cc:504] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:39.398948 32527 leader_election.cc:288] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33 [CANDIDATE]: Term 1 election: Requested vote from peers fa1a7cdb80ba44c48f80a60e6627f03b (127.29.78.129:34751), 2d70d72ef23146c7b87134cbd485e85a (127.29.78.131:43313)
I20260808 18:28:39.399288 32411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cc9eb66eb6e4122a932da1c85843889" candidate_uuid: "d7b686b608e248cbb46aa83e75772a33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d70d72ef23146c7b87134cbd485e85a"
I20260808 18:28:39.399293 32109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cc9eb66eb6e4122a932da1c85843889" candidate_uuid: "d7b686b608e248cbb46aa83e75772a33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b"
I20260808 18:28:39.399384 32411 raft_consensus.cc:3037] T 0cc9eb66eb6e4122a932da1c85843889 P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:39.399461 32109 raft_consensus.cc:3037] T 0cc9eb66eb6e4122a932da1c85843889 P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:39.400388 32411 raft_consensus.cc:2454] T 0cc9eb66eb6e4122a932da1c85843889 P 2d70d72ef23146c7b87134cbd485e85a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b686b608e248cbb46aa83e75772a33 in term 1.
I20260808 18:28:39.400689 32188 leader_election.cc:302] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d70d72ef23146c7b87134cbd485e85a, d7b686b608e248cbb46aa83e75772a33; no voters:
I20260808 18:28:39.400872 32527 raft_consensus.cc:2781] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:39.400992 32527 raft_consensus.cc:686] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33 [term 1 LEADER]: Becoming Leader. State: Replica: d7b686b608e248cbb46aa83e75772a33, State: Running, Role: LEADER
I20260808 18:28:39.401137 32527 consensus_queue.cc:237] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:39.401206 32109 raft_consensus.cc:2454] T 0cc9eb66eb6e4122a932da1c85843889 P fa1a7cdb80ba44c48f80a60e6627f03b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b686b608e248cbb46aa83e75772a33 in term 1.
I20260808 18:28:39.402923 31928 catalog_manager.cc:5495] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7b686b608e248cbb46aa83e75772a33 (127.29.78.130). New cstate: current_term: 1 leader_uuid: "d7b686b608e248cbb46aa83e75772a33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } health_report { overall_health: UNKNOWN } } }
I20260808 18:28:39.420238 32498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:39.416952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.29.78.130:44535)
0808 18:28:39.417081 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:39.417085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:39.417127 (+ 42us) server_negotiation.cc:407] Connection header received
0808 18:28:39.417289 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:39.417292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:39.417341 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:39.417419 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:28:39.417662 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.418170 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.419214 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.419439 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.419544 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:39.419804 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:28:39.419877 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:28:39.419955 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:39.420049 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 18:28:39.420099 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 18:28:39.428687 32557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:39.425882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44535 (local address 127.29.78.130:44865)
0808 18:28:39.426030 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:39.426043 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:39.426104 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:39.426291 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:39.426293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:39.426298 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 18:28:39.426395 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:39.426446 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.427179 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.427181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:39.427728 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:39.427735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.428148 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.428151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:39.428208 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:39.428230 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 18:28:39.428464 (+ 234us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 18:28:39.428468 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 18:28:39.428521 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 18:28:39.428578 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":16}
I20260808 18:28:39.428689 32498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:39.425988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:44865 (local address 127.29.78.130:44535)
0808 18:28:39.426099 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:39.426106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:39.426126 (+ 20us) server_negotiation.cc:407] Connection header received
0808 18:28:39.426162 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:39.426164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:39.426203 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:39.426276 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:28:39.426538 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.427087 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.427826 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:39.428054 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:39.428133 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:39.428328 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:28:39.428393 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:28:39.428451 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:39.428529 (+ 78us) server_negotiation.cc:299] Negotiation successful
0808 18:28:39.428567 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":43}
I20260808 18:28:39.434444 32411 raft_consensus.cc:1261] T 0cc9eb66eb6e4122a932da1c85843889 P 2d70d72ef23146c7b87134cbd485e85a [term 1 FOLLOWER]: Refusing update from remote peer d7b686b608e248cbb46aa83e75772a33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:28:39.434531 32109 raft_consensus.cc:1261] T 0cc9eb66eb6e4122a932da1c85843889 P fa1a7cdb80ba44c48f80a60e6627f03b [term 1 FOLLOWER]: Refusing update from remote peer d7b686b608e248cbb46aa83e75772a33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:28:39.434840 32545 consensus_queue.cc:1035] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:28:39.435036 32567 consensus_queue.cc:1035] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 18:28:39.449607 32535 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 18:28:41.004366 31927 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:28:41.004537 31927 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:28:41.007063 31927 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0: Sending DeleteTablet for 3 replicas of tablet 0cc9eb66eb6e4122a932da1c85843889
I20260808 18:28:41.007674 32245 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cc9eb66eb6e4122a932da1c85843889 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:41 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:53796
I20260808 18:28:41.008066 32593 tablet_replica.cc:330] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33: stopping tablet replica
I20260808 18:28:41.008042 32083 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cc9eb66eb6e4122a932da1c85843889 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:41 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:47270
I20260808 18:28:41.008219 32593 raft_consensus.cc:2229] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:28:41.008440 32595 tablet_replica.cc:330] T 0cc9eb66eb6e4122a932da1c85843889 P fa1a7cdb80ba44c48f80a60e6627f03b: stopping tablet replica
I20260808 18:28:41.008535 32593 raft_consensus.cc:2258] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:28:41.008683 32595 raft_consensus.cc:2229] T 0cc9eb66eb6e4122a932da1c85843889 P fa1a7cdb80ba44c48f80a60e6627f03b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:28:41.008813 32595 raft_consensus.cc:2258] T 0cc9eb66eb6e4122a932da1c85843889 P fa1a7cdb80ba44c48f80a60e6627f03b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:28:41.009562 32595 ts_tablet_manager.cc:1905] T 0cc9eb66eb6e4122a932da1c85843889 P fa1a7cdb80ba44c48f80a60e6627f03b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:28:41.012212 32398 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cc9eb66eb6e4122a932da1c85843889 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:41 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:33218
I20260808 18:28:41.012615 32597 tablet_replica.cc:330] T 0cc9eb66eb6e4122a932da1c85843889 P 2d70d72ef23146c7b87134cbd485e85a: stopping tablet replica
I20260808 18:28:41.012822 32597 raft_consensus.cc:2229] T 0cc9eb66eb6e4122a932da1c85843889 P 2d70d72ef23146c7b87134cbd485e85a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:28:41.012856 32595 ts_tablet_manager.cc:1918] T 0cc9eb66eb6e4122a932da1c85843889 P fa1a7cdb80ba44c48f80a60e6627f03b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 18:28:41.012943 32595 log.cc:1196] T 0cc9eb66eb6e4122a932da1c85843889 P fa1a7cdb80ba44c48f80a60e6627f03b: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-0/wal/wals/0cc9eb66eb6e4122a932da1c85843889
I20260808 18:28:41.013087 32597 raft_consensus.cc:2258] T 0cc9eb66eb6e4122a932da1c85843889 P 2d70d72ef23146c7b87134cbd485e85a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:28:41.013104 32593 ts_tablet_manager.cc:1905] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:28:41.013270 32595 ts_tablet_manager.cc:1939] T 0cc9eb66eb6e4122a932da1c85843889 P fa1a7cdb80ba44c48f80a60e6627f03b: Deleting consensus metadata
I20260808 18:28:41.013939 32597 ts_tablet_manager.cc:1905] T 0cc9eb66eb6e4122a932da1c85843889 P 2d70d72ef23146c7b87134cbd485e85a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:28:41.014043 31910 catalog_manager.cc:4841] TS fa1a7cdb80ba44c48f80a60e6627f03b (127.29.78.129:34751): tablet 0cc9eb66eb6e4122a932da1c85843889 (table test-table [id=403768197c15454d8873945894912445]) successfully deleted
I20260808 18:28:41.016324 32593 ts_tablet_manager.cc:1918] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 18:28:41.016428 32593 log.cc:1196] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/0cc9eb66eb6e4122a932da1c85843889
I20260808 18:28:41.016539 32597 ts_tablet_manager.cc:1918] T 0cc9eb66eb6e4122a932da1c85843889 P 2d70d72ef23146c7b87134cbd485e85a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 18:28:41.016637 32597 log.cc:1196] T 0cc9eb66eb6e4122a932da1c85843889 P 2d70d72ef23146c7b87134cbd485e85a: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/0cc9eb66eb6e4122a932da1c85843889
I20260808 18:28:41.016793 32593 ts_tablet_manager.cc:1939] T 0cc9eb66eb6e4122a932da1c85843889 P d7b686b608e248cbb46aa83e75772a33: Deleting consensus metadata
I20260808 18:28:41.016924 32597 ts_tablet_manager.cc:1939] T 0cc9eb66eb6e4122a932da1c85843889 P 2d70d72ef23146c7b87134cbd485e85a: Deleting consensus metadata
I20260808 18:28:41.017567 31910 catalog_manager.cc:4841] TS d7b686b608e248cbb46aa83e75772a33 (127.29.78.130:44535): tablet 0cc9eb66eb6e4122a932da1c85843889 (table test-table [id=403768197c15454d8873945894912445]) successfully deleted
I20260808 18:28:41.017614 31911 catalog_manager.cc:4841] TS 2d70d72ef23146c7b87134cbd485e85a (127.29.78.131:43313): tablet 0cc9eb66eb6e4122a932da1c85843889 (table test-table [id=403768197c15454d8873945894912445]) successfully deleted
Aug 08 18:28:41 dist-test-slave-gp6n krb5kdc[31877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213721, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 18:28:41 dist-test-slave-gp6n krb5kdc[31877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213721, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:41.038072 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:41.029095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56880 (local address 127.29.78.190:32829)
0808 18:28:41.029387 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:41.029396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:41.029497 (+ 101us) server_negotiation.cc:407] Connection header received
0808 18:28:41.029686 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:41.029689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:41.029745 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:41.029850 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:41.030657 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:41.031216 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:41.032100 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:41.032332 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:41.032440 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:41.034913 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:41.034916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:41.034918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:41.034942 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:41.036487 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:41.036880 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:41.036882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:41.036884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:41.036960 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:41.037243 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:41.037245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:41.037247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:41.037419 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:41.037573 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:41.037735 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 18:28:41.037840 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":97,"threads_started":1}
I20260808 18:28:41.040843 31927 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:28:41.041290 31927 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:28:41.046167 32245 tablet_service.cc:1467] Processing CreateTablet for tablet fb5f9d3ed6bb44d794b786681c3cdcdf (DEFAULT_TABLE table=test-table [id=a5adfea151eb4d26aee221bc338db456]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:41.046166 32083 tablet_service.cc:1467] Processing CreateTablet for tablet fb5f9d3ed6bb44d794b786681c3cdcdf (DEFAULT_TABLE table=test-table [id=a5adfea151eb4d26aee221bc338db456]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:41.046238 32398 tablet_service.cc:1467] Processing CreateTablet for tablet fb5f9d3ed6bb44d794b786681c3cdcdf (DEFAULT_TABLE table=test-table [id=a5adfea151eb4d26aee221bc338db456]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:41.046548 32245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb5f9d3ed6bb44d794b786681c3cdcdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:41.046576 32083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb5f9d3ed6bb44d794b786681c3cdcdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:41.046579 32398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb5f9d3ed6bb44d794b786681c3cdcdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:41.048707 32605 tablet_bootstrap.cc:492] T fb5f9d3ed6bb44d794b786681c3cdcdf P d7b686b608e248cbb46aa83e75772a33: Bootstrap starting.
I20260808 18:28:41.049232 32606 tablet_bootstrap.cc:492] T fb5f9d3ed6bb44d794b786681c3cdcdf P fa1a7cdb80ba44c48f80a60e6627f03b: Bootstrap starting.
I20260808 18:28:41.049541 32607 tablet_bootstrap.cc:492] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a: Bootstrap starting.
I20260808 18:28:41.050096 32605 tablet_bootstrap.cc:654] T fb5f9d3ed6bb44d794b786681c3cdcdf P d7b686b608e248cbb46aa83e75772a33: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:41.050184 32606 tablet_bootstrap.cc:654] T fb5f9d3ed6bb44d794b786681c3cdcdf P fa1a7cdb80ba44c48f80a60e6627f03b: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:41.050549 32607 tablet_bootstrap.cc:654] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:41.051352 32605 tablet_bootstrap.cc:492] T fb5f9d3ed6bb44d794b786681c3cdcdf P d7b686b608e248cbb46aa83e75772a33: No bootstrap required, opened a new log
I20260808 18:28:41.051424 32605 ts_tablet_manager.cc:1397] T fb5f9d3ed6bb44d794b786681c3cdcdf P d7b686b608e248cbb46aa83e75772a33: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260808 18:28:41.051652 32606 tablet_bootstrap.cc:492] T fb5f9d3ed6bb44d794b786681c3cdcdf P fa1a7cdb80ba44c48f80a60e6627f03b: No bootstrap required, opened a new log
I20260808 18:28:41.051755 32606 ts_tablet_manager.cc:1397] T fb5f9d3ed6bb44d794b786681c3cdcdf P fa1a7cdb80ba44c48f80a60e6627f03b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:28:41.051914 32605 raft_consensus.cc:348] T fb5f9d3ed6bb44d794b786681c3cdcdf P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:41.052073 32605 raft_consensus.cc:374] T fb5f9d3ed6bb44d794b786681c3cdcdf P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:41.052119 32605 raft_consensus.cc:729] T fb5f9d3ed6bb44d794b786681c3cdcdf P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b686b608e248cbb46aa83e75772a33, State: Initialized, Role: FOLLOWER
I20260808 18:28:41.052197 32606 raft_consensus.cc:348] T fb5f9d3ed6bb44d794b786681c3cdcdf P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:41.052248 32605 consensus_queue.cc:260] T fb5f9d3ed6bb44d794b786681c3cdcdf P d7b686b608e248cbb46aa83e75772a33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:41.052320 32606 raft_consensus.cc:374] T fb5f9d3ed6bb44d794b786681c3cdcdf P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:41.052367 32606 raft_consensus.cc:729] T fb5f9d3ed6bb44d794b786681c3cdcdf P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1a7cdb80ba44c48f80a60e6627f03b, State: Initialized, Role: FOLLOWER
I20260808 18:28:41.052557 32605 ts_tablet_manager.cc:1428] T fb5f9d3ed6bb44d794b786681c3cdcdf P d7b686b608e248cbb46aa83e75772a33: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260808 18:28:41.052507 32606 consensus_queue.cc:260] T fb5f9d3ed6bb44d794b786681c3cdcdf P fa1a7cdb80ba44c48f80a60e6627f03b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:41.052938 32606 ts_tablet_manager.cc:1428] T fb5f9d3ed6bb44d794b786681c3cdcdf P fa1a7cdb80ba44c48f80a60e6627f03b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:28:41.053458 32607 tablet_bootstrap.cc:492] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a: No bootstrap required, opened a new log
I20260808 18:28:41.053542 32607 ts_tablet_manager.cc:1397] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260808 18:28:41.053902 32607 raft_consensus.cc:348] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:41.054016 32607 raft_consensus.cc:374] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:41.054062 32607 raft_consensus.cc:729] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d70d72ef23146c7b87134cbd485e85a, State: Initialized, Role: FOLLOWER
I20260808 18:28:41.054194 32607 consensus_queue.cc:260] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:41.054623 32607 ts_tablet_manager.cc:1428] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 18:28:41.143864 32613 raft_consensus.cc:482] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:28:41.144032 32613 raft_consensus.cc:504] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:41.144524 32613 leader_election.cc:288] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa1a7cdb80ba44c48f80a60e6627f03b (127.29.78.129:34751), d7b686b608e248cbb46aa83e75772a33 (127.29.78.130:44535)
I20260808 18:28:41.145035 32109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5f9d3ed6bb44d794b786681c3cdcdf" candidate_uuid: "2d70d72ef23146c7b87134cbd485e85a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" is_pre_election: true
I20260808 18:28:41.145026 32267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5f9d3ed6bb44d794b786681c3cdcdf" candidate_uuid: "2d70d72ef23146c7b87134cbd485e85a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b686b608e248cbb46aa83e75772a33" is_pre_election: true
I20260808 18:28:41.145195 32109 raft_consensus.cc:2454] T fb5f9d3ed6bb44d794b786681c3cdcdf P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d70d72ef23146c7b87134cbd485e85a in term 0.
I20260808 18:28:41.145195 32267 raft_consensus.cc:2454] T fb5f9d3ed6bb44d794b786681c3cdcdf P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d70d72ef23146c7b87134cbd485e85a in term 0.
I20260808 18:28:41.145555 32344 leader_election.cc:302] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d70d72ef23146c7b87134cbd485e85a, fa1a7cdb80ba44c48f80a60e6627f03b; no voters:
I20260808 18:28:41.145751 32613 raft_consensus.cc:2781] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:28:41.145813 32613 raft_consensus.cc:482] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:28:41.145852 32613 raft_consensus.cc:3037] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:41.147313 32613 raft_consensus.cc:504] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:41.147691 32613 leader_election.cc:288] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a [CANDIDATE]: Term 1 election: Requested vote from peers fa1a7cdb80ba44c48f80a60e6627f03b (127.29.78.129:34751), d7b686b608e248cbb46aa83e75772a33 (127.29.78.130:44535)
I20260808 18:28:41.148101 32109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5f9d3ed6bb44d794b786681c3cdcdf" candidate_uuid: "2d70d72ef23146c7b87134cbd485e85a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b"
I20260808 18:28:41.148101 32267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5f9d3ed6bb44d794b786681c3cdcdf" candidate_uuid: "2d70d72ef23146c7b87134cbd485e85a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b686b608e248cbb46aa83e75772a33"
I20260808 18:28:41.148248 32109 raft_consensus.cc:3037] T fb5f9d3ed6bb44d794b786681c3cdcdf P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:41.148248 32267 raft_consensus.cc:3037] T fb5f9d3ed6bb44d794b786681c3cdcdf P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:41.149456 32267 raft_consensus.cc:2454] T fb5f9d3ed6bb44d794b786681c3cdcdf P d7b686b608e248cbb46aa83e75772a33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d70d72ef23146c7b87134cbd485e85a in term 1.
I20260808 18:28:41.149456 32109 raft_consensus.cc:2454] T fb5f9d3ed6bb44d794b786681c3cdcdf P fa1a7cdb80ba44c48f80a60e6627f03b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d70d72ef23146c7b87134cbd485e85a in term 1.
I20260808 18:28:41.149783 32344 leader_election.cc:302] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d70d72ef23146c7b87134cbd485e85a, d7b686b608e248cbb46aa83e75772a33; no voters:
I20260808 18:28:41.149979 32613 raft_consensus.cc:2781] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:41.150260 32613 raft_consensus.cc:686] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a [term 1 LEADER]: Becoming Leader. State: Replica: 2d70d72ef23146c7b87134cbd485e85a, State: Running, Role: LEADER
I20260808 18:28:41.150506 32613 consensus_queue.cc:237] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:41.152204 31928 catalog_manager.cc:5495] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d70d72ef23146c7b87134cbd485e85a (127.29.78.131). New cstate: current_term: 1 leader_uuid: "2d70d72ef23146c7b87134cbd485e85a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } health_report { overall_health: HEALTHY } } }
I20260808 18:28:41.183192 32618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:41.180144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33240 (local address 127.29.78.131:43313)
0808 18:28:41.180439 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:41.180446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:41.180458 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:28:41.180496 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:41.180498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:41.180538 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:41.180611 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:28:41.180849 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:41.181384 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:41.182193 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:41.182461 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:41.182543 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:41.182758 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:28:41.182833 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:28:41.182887 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:41.182966 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 18:28:41.183008 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":155,"threads_started":1}
I20260808 18:28:41.189721 32267 raft_consensus.cc:1261] T fb5f9d3ed6bb44d794b786681c3cdcdf P d7b686b608e248cbb46aa83e75772a33 [term 1 FOLLOWER]: Refusing update from remote peer 2d70d72ef23146c7b87134cbd485e85a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:28:41.189735 32109 raft_consensus.cc:1261] T fb5f9d3ed6bb44d794b786681c3cdcdf P fa1a7cdb80ba44c48f80a60e6627f03b [term 1 FOLLOWER]: Refusing update from remote peer 2d70d72ef23146c7b87134cbd485e85a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:28:41.190182 32614 consensus_queue.cc:1035] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:28:41.190315 32613 consensus_queue.cc:1035] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:28:41.222533 31928 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:28:41.222678 31928 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:28:41.224560 31928 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0: Sending DeleteTablet for 3 replicas of tablet fb5f9d3ed6bb44d794b786681c3cdcdf
I20260808 18:28:41.225116 32083 tablet_service.cc:1514] Processing DeleteTablet for tablet fb5f9d3ed6bb44d794b786681c3cdcdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:41 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:47270
I20260808 18:28:41.225152 32245 tablet_service.cc:1514] Processing DeleteTablet for tablet fb5f9d3ed6bb44d794b786681c3cdcdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:41 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:53796
I20260808 18:28:41.225142 32398 tablet_service.cc:1514] Processing DeleteTablet for tablet fb5f9d3ed6bb44d794b786681c3cdcdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:41 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:33218
I20260808 18:28:41.225302 32597 tablet_replica.cc:330] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a: stopping tablet replica
I20260808 18:28:41.225313 32595 tablet_replica.cc:330] T fb5f9d3ed6bb44d794b786681c3cdcdf P fa1a7cdb80ba44c48f80a60e6627f03b: stopping tablet replica
I20260808 18:28:41.225417 32597 raft_consensus.cc:2229] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:28:41.225428 32595 raft_consensus.cc:2229] T fb5f9d3ed6bb44d794b786681c3cdcdf P fa1a7cdb80ba44c48f80a60e6627f03b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:28:41.225566 32595 raft_consensus.cc:2258] T fb5f9d3ed6bb44d794b786681c3cdcdf P fa1a7cdb80ba44c48f80a60e6627f03b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:28:41.225618 32597 raft_consensus.cc:2258] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:28:41.225754 32593 tablet_replica.cc:330] T fb5f9d3ed6bb44d794b786681c3cdcdf P d7b686b608e248cbb46aa83e75772a33: stopping tablet replica
I20260808 18:28:41.225908 32593 raft_consensus.cc:2229] T fb5f9d3ed6bb44d794b786681c3cdcdf P d7b686b608e248cbb46aa83e75772a33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:28:41.225857 31928 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 18:28:41.226022 32593 raft_consensus.cc:2258] T fb5f9d3ed6bb44d794b786681c3cdcdf P d7b686b608e248cbb46aa83e75772a33 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 18:28:41.226269 31928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:28:41.226351 32597 ts_tablet_manager.cc:1905] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:28:41.226850 32595 ts_tablet_manager.cc:1905] T fb5f9d3ed6bb44d794b786681c3cdcdf P fa1a7cdb80ba44c48f80a60e6627f03b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:28:41.226843 32593 ts_tablet_manager.cc:1905] T fb5f9d3ed6bb44d794b786681c3cdcdf P d7b686b608e248cbb46aa83e75772a33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:28:41.228585 32597 ts_tablet_manager.cc:1918] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:28:41.228639 32597 log.cc:1196] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/fb5f9d3ed6bb44d794b786681c3cdcdf
I20260808 18:28:41.228837 32597 ts_tablet_manager.cc:1939] T fb5f9d3ed6bb44d794b786681c3cdcdf P 2d70d72ef23146c7b87134cbd485e85a: Deleting consensus metadata
I20260808 18:28:41.229373 32595 ts_tablet_manager.cc:1918] T fb5f9d3ed6bb44d794b786681c3cdcdf P fa1a7cdb80ba44c48f80a60e6627f03b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:28:41.229415 32593 ts_tablet_manager.cc:1918] T fb5f9d3ed6bb44d794b786681c3cdcdf P d7b686b608e248cbb46aa83e75772a33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:28:41.229445 32595 log.cc:1196] T fb5f9d3ed6bb44d794b786681c3cdcdf P fa1a7cdb80ba44c48f80a60e6627f03b: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-0/wal/wals/fb5f9d3ed6bb44d794b786681c3cdcdf
I20260808 18:28:41.229483 32593 log.cc:1196] T fb5f9d3ed6bb44d794b786681c3cdcdf P d7b686b608e248cbb46aa83e75772a33: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/fb5f9d3ed6bb44d794b786681c3cdcdf
I20260808 18:28:41.229722 32595 ts_tablet_manager.cc:1939] T fb5f9d3ed6bb44d794b786681c3cdcdf P fa1a7cdb80ba44c48f80a60e6627f03b: Deleting consensus metadata
I20260808 18:28:41.229753 32593 ts_tablet_manager.cc:1939] T fb5f9d3ed6bb44d794b786681c3cdcdf P d7b686b608e248cbb46aa83e75772a33: Deleting consensus metadata
I20260808 18:28:41.230389 31910 catalog_manager.cc:4841] TS fa1a7cdb80ba44c48f80a60e6627f03b (127.29.78.129:34751): tablet fb5f9d3ed6bb44d794b786681c3cdcdf (table test-table [id=a5adfea151eb4d26aee221bc338db456]) successfully deleted
I20260808 18:28:41.230428 31911 catalog_manager.cc:4841] TS 2d70d72ef23146c7b87134cbd485e85a (127.29.78.131:43313): tablet fb5f9d3ed6bb44d794b786681c3cdcdf (table test-table [id=a5adfea151eb4d26aee221bc338db456]) successfully deleted
I20260808 18:28:41.231009 31910 catalog_manager.cc:4841] TS d7b686b608e248cbb46aa83e75772a33 (127.29.78.130:44535): tablet fb5f9d3ed6bb44d794b786681c3cdcdf (table test-table [id=a5adfea151eb4d26aee221bc338db456]) successfully deleted
I20260808 18:28:41.232828 32083 tablet_service.cc:1467] Processing CreateTablet for tablet 621e921ecf90436d9367dce7e2016fcc (DEFAULT_TABLE table=test-table [id=79929ada254b4c41920697d0542dac75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:41.232854 32245 tablet_service.cc:1467] Processing CreateTablet for tablet 621e921ecf90436d9367dce7e2016fcc (DEFAULT_TABLE table=test-table [id=79929ada254b4c41920697d0542dac75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:41.232993 32398 tablet_service.cc:1467] Processing CreateTablet for tablet 621e921ecf90436d9367dce7e2016fcc (DEFAULT_TABLE table=test-table [id=79929ada254b4c41920697d0542dac75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:41.233078 32083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 621e921ecf90436d9367dce7e2016fcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:41.233083 32245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 621e921ecf90436d9367dce7e2016fcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:41.233245 32398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 621e921ecf90436d9367dce7e2016fcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:41.234961 32607 tablet_bootstrap.cc:492] T 621e921ecf90436d9367dce7e2016fcc P 2d70d72ef23146c7b87134cbd485e85a: Bootstrap starting.
I20260808 18:28:41.235248 32606 tablet_bootstrap.cc:492] T 621e921ecf90436d9367dce7e2016fcc P fa1a7cdb80ba44c48f80a60e6627f03b: Bootstrap starting.
I20260808 18:28:41.235498 32605 tablet_bootstrap.cc:492] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33: Bootstrap starting.
I20260808 18:28:41.235751 32607 tablet_bootstrap.cc:654] T 621e921ecf90436d9367dce7e2016fcc P 2d70d72ef23146c7b87134cbd485e85a: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:41.236166 32606 tablet_bootstrap.cc:654] T 621e921ecf90436d9367dce7e2016fcc P fa1a7cdb80ba44c48f80a60e6627f03b: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:41.236270 32605 tablet_bootstrap.cc:654] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:41.236788 32607 tablet_bootstrap.cc:492] T 621e921ecf90436d9367dce7e2016fcc P 2d70d72ef23146c7b87134cbd485e85a: No bootstrap required, opened a new log
I20260808 18:28:41.236886 32607 ts_tablet_manager.cc:1397] T 621e921ecf90436d9367dce7e2016fcc P 2d70d72ef23146c7b87134cbd485e85a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260808 18:28:41.237367 32606 tablet_bootstrap.cc:492] T 621e921ecf90436d9367dce7e2016fcc P fa1a7cdb80ba44c48f80a60e6627f03b: No bootstrap required, opened a new log
I20260808 18:28:41.237298 32607 raft_consensus.cc:348] T 621e921ecf90436d9367dce7e2016fcc P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:41.237406 32607 raft_consensus.cc:374] T 621e921ecf90436d9367dce7e2016fcc P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:41.237442 32606 ts_tablet_manager.cc:1397] T 621e921ecf90436d9367dce7e2016fcc P fa1a7cdb80ba44c48f80a60e6627f03b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 18:28:41.237437 32607 raft_consensus.cc:729] T 621e921ecf90436d9367dce7e2016fcc P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d70d72ef23146c7b87134cbd485e85a, State: Initialized, Role: FOLLOWER
I20260808 18:28:41.237592 32607 consensus_queue.cc:260] T 621e921ecf90436d9367dce7e2016fcc P 2d70d72ef23146c7b87134cbd485e85a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:41.237696 32605 tablet_bootstrap.cc:492] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33: No bootstrap required, opened a new log
I20260808 18:28:41.237795 32605 ts_tablet_manager.cc:1397] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260808 18:28:41.237852 32607 ts_tablet_manager.cc:1428] T 621e921ecf90436d9367dce7e2016fcc P 2d70d72ef23146c7b87134cbd485e85a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 18:28:41.237862 32606 raft_consensus.cc:348] T 621e921ecf90436d9367dce7e2016fcc P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:41.237959 32606 raft_consensus.cc:374] T 621e921ecf90436d9367dce7e2016fcc P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:41.237998 32606 raft_consensus.cc:729] T 621e921ecf90436d9367dce7e2016fcc P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1a7cdb80ba44c48f80a60e6627f03b, State: Initialized, Role: FOLLOWER
I20260808 18:28:41.238140 32606 consensus_queue.cc:260] T 621e921ecf90436d9367dce7e2016fcc P fa1a7cdb80ba44c48f80a60e6627f03b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:41.238229 32605 raft_consensus.cc:348] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:41.238369 32605 raft_consensus.cc:374] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:41.238456 32605 raft_consensus.cc:729] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b686b608e248cbb46aa83e75772a33, State: Initialized, Role: FOLLOWER
I20260808 18:28:41.238464 32606 ts_tablet_manager.cc:1428] T 621e921ecf90436d9367dce7e2016fcc P fa1a7cdb80ba44c48f80a60e6627f03b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:28:41.238595 32605 consensus_queue.cc:260] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:41.238914 32605 ts_tablet_manager.cc:1428] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:28:41.445785 32581 raft_consensus.cc:482] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:28:41.445990 32581 raft_consensus.cc:504] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:41.446637 32581 leader_election.cc:288] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa1a7cdb80ba44c48f80a60e6627f03b (127.29.78.129:34751), 2d70d72ef23146c7b87134cbd485e85a (127.29.78.131:43313)
I20260808 18:28:41.447137 32109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "621e921ecf90436d9367dce7e2016fcc" candidate_uuid: "d7b686b608e248cbb46aa83e75772a33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" is_pre_election: true
I20260808 18:28:41.447137 32411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "621e921ecf90436d9367dce7e2016fcc" candidate_uuid: "d7b686b608e248cbb46aa83e75772a33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d70d72ef23146c7b87134cbd485e85a" is_pre_election: true
I20260808 18:28:41.447296 32411 raft_consensus.cc:2454] T 621e921ecf90436d9367dce7e2016fcc P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7b686b608e248cbb46aa83e75772a33 in term 0.
I20260808 18:28:41.447296 32109 raft_consensus.cc:2454] T 621e921ecf90436d9367dce7e2016fcc P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7b686b608e248cbb46aa83e75772a33 in term 0.
I20260808 18:28:41.447635 32187 leader_election.cc:302] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2d70d72ef23146c7b87134cbd485e85a, d7b686b608e248cbb46aa83e75772a33, fa1a7cdb80ba44c48f80a60e6627f03b; no voters:
I20260808 18:28:41.447788 32581 raft_consensus.cc:2781] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:28:41.447844 32581 raft_consensus.cc:482] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:28:41.447887 32581 raft_consensus.cc:3037] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:41.449057 32581 raft_consensus.cc:504] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:41.449388 32581 leader_election.cc:288] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33 [CANDIDATE]: Term 1 election: Requested vote from peers fa1a7cdb80ba44c48f80a60e6627f03b (127.29.78.129:34751), 2d70d72ef23146c7b87134cbd485e85a (127.29.78.131:43313)
I20260808 18:28:41.449704 32109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "621e921ecf90436d9367dce7e2016fcc" candidate_uuid: "d7b686b608e248cbb46aa83e75772a33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b"
I20260808 18:28:41.449831 32109 raft_consensus.cc:3037] T 621e921ecf90436d9367dce7e2016fcc P fa1a7cdb80ba44c48f80a60e6627f03b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:41.449927 32411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "621e921ecf90436d9367dce7e2016fcc" candidate_uuid: "d7b686b608e248cbb46aa83e75772a33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d70d72ef23146c7b87134cbd485e85a"
I20260808 18:28:41.450105 32411 raft_consensus.cc:3037] T 621e921ecf90436d9367dce7e2016fcc P 2d70d72ef23146c7b87134cbd485e85a [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:41.450930 32109 raft_consensus.cc:2454] T 621e921ecf90436d9367dce7e2016fcc P fa1a7cdb80ba44c48f80a60e6627f03b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b686b608e248cbb46aa83e75772a33 in term 1.
I20260808 18:28:41.451243 32187 leader_election.cc:302] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7b686b608e248cbb46aa83e75772a33, fa1a7cdb80ba44c48f80a60e6627f03b; no voters:
I20260808 18:28:41.451406 32581 raft_consensus.cc:2781] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:41.451509 32581 raft_consensus.cc:686] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33 [term 1 LEADER]: Becoming Leader. State: Replica: d7b686b608e248cbb46aa83e75772a33, State: Running, Role: LEADER
I20260808 18:28:41.451524 32411 raft_consensus.cc:2454] T 621e921ecf90436d9367dce7e2016fcc P 2d70d72ef23146c7b87134cbd485e85a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b686b608e248cbb46aa83e75772a33 in term 1.
I20260808 18:28:41.451649 32581 consensus_queue.cc:237] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } }
I20260808 18:28:41.453382 31927 catalog_manager.cc:5495] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7b686b608e248cbb46aa83e75772a33 (127.29.78.130). New cstate: current_term: 1 leader_uuid: "d7b686b608e248cbb46aa83e75772a33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b686b608e248cbb46aa83e75772a33" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 } health_report { overall_health: UNKNOWN } } }
I20260808 18:28:41.485296 32633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:41.481721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45562 (local address 127.29.78.130:44535)
0808 18:28:41.482025 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:41.482033 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:41.482045 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:28:41.482089 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:41.482092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:41.482135 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:41.482239 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:28:41.482563 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:41.483161 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:41.484098 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:41.484375 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:41.484500 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:41.484792 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:28:41.484873 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:28:41.484965 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:41.485070 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 18:28:41.485125 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":110,"threads_started":1}
I20260808 18:28:41.491950 32109 raft_consensus.cc:1261] T 621e921ecf90436d9367dce7e2016fcc P fa1a7cdb80ba44c48f80a60e6627f03b [term 1 FOLLOWER]: Refusing update from remote peer d7b686b608e248cbb46aa83e75772a33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:28:41.491995 32411 raft_consensus.cc:1261] T 621e921ecf90436d9367dce7e2016fcc P 2d70d72ef23146c7b87134cbd485e85a [term 1 FOLLOWER]: Refusing update from remote peer d7b686b608e248cbb46aa83e75772a33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:28:41.492422 32581 consensus_queue.cc:1035] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d70d72ef23146c7b87134cbd485e85a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:28:41.492650 32576 consensus_queue.cc:1035] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa1a7cdb80ba44c48f80a60e6627f03b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 18:28:41.508031 32574 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 18:28:43.060869 31929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:28:43.061026 31929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:28:43.063426 31929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7540f6de5b0c4525a2bcb20ae04ab9c0: Sending DeleteTablet for 3 replicas of tablet 621e921ecf90436d9367dce7e2016fcc
I20260808 18:28:43.064072 32398 tablet_service.cc:1514] Processing DeleteTablet for tablet 621e921ecf90436d9367dce7e2016fcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:43 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:33218
I20260808 18:28:43.064116 32083 tablet_service.cc:1514] Processing DeleteTablet for tablet 621e921ecf90436d9367dce7e2016fcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:43 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:47270
I20260808 18:28:43.064464 32659 tablet_replica.cc:330] T 621e921ecf90436d9367dce7e2016fcc P 2d70d72ef23146c7b87134cbd485e85a: stopping tablet replica
I20260808 18:28:43.064500 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 32006
I20260808 18:28:43.064587 32245 tablet_service.cc:1514] Processing DeleteTablet for tablet 621e921ecf90436d9367dce7e2016fcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:43 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:53796
I20260808 18:28:43.067480 32659 raft_consensus.cc:2229] T 621e921ecf90436d9367dce7e2016fcc P 2d70d72ef23146c7b87134cbd485e85a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:28:43.067646 32659 raft_consensus.cc:2258] T 621e921ecf90436d9367dce7e2016fcc P 2d70d72ef23146c7b87134cbd485e85a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:28:43.068449 32659 ts_tablet_manager.cc:1905] T 621e921ecf90436d9367dce7e2016fcc P 2d70d72ef23146c7b87134cbd485e85a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:28:43.069015 32662 tablet_replica.cc:330] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33: stopping tablet replica
I20260808 18:28:43.069161 32662 raft_consensus.cc:2229] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:28:43.069382 32662 raft_consensus.cc:2258] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:28:43.069881 32662 ts_tablet_manager.cc:1905] T 621e921ecf90436d9367dce7e2016fcc P d7b686b608e248cbb46aa83e75772a33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 18:28:43.074924 31910 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:34751 (error 108)
I20260808 18:28:43.075189 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 32169
I20260808 18:28:43.078847 32664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:43.078321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:34751 (local address 127.0.0.1:46868)
0808 18:28:43.078684 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:43.078758 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:34751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":131,"threads_started":1}
W20260808 18:28:43.078996 31910 catalog_manager.cc:4584] TS fa1a7cdb80ba44c48f80a60e6627f03b (127.29.78.129:34751): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 621e921ecf90436d9367dce7e2016fcc: Network error: Client connection negotiation failed: client connection to 127.29.78.129:34751: connect: Connection refused (error 111)
I20260808 18:28:43.132512 32664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:43.132123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:34751 (local address 127.0.0.1:46870)
0808 18:28:43.132336 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:43.132414 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:34751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260808 18:28:43.133441 32664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:43.133179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:34751 (local address 127.0.0.1:46878)
0808 18:28:43.133323 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:43.133376 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:34751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 18:28:43.157632 32659 ts_tablet_manager.cc:1918] T 621e921ecf90436d9367dce7e2016fcc P 2d70d72ef23146c7b87134cbd485e85a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 18:28:43.157799 32659 log.cc:1196] T 621e921ecf90436d9367dce7e2016fcc P 2d70d72ef23146c7b87134cbd485e85a: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/621e921ecf90436d9367dce7e2016fcc
I20260808 18:28:43.158083 32659 ts_tablet_manager.cc:1939] T 621e921ecf90436d9367dce7e2016fcc P 2d70d72ef23146c7b87134cbd485e85a: Deleting consensus metadata
I20260808 18:28:43.163795 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 32327
I20260808 18:28:43.166343 32664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:43.163898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44535 (local address 127.0.0.1:45572)
0808 18:28:43.166084 (+ 2186us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:43.166198 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.130:44535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2064}
I20260808 18:28:43.188021 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 31891
2026-08-08T18:28:43Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6835 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:28:44 dist-test-slave-gp6n krb5kdc[32667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:28:44 dist-test-slave-gp6n krb5kdc[32667](info): set up 2 sockets
Aug 08 18:28:44 dist-test-slave-gp6n krb5kdc[32667](info): commencing operation
krb5kdc: starting...
W20260808 18:28:45.197062 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.983s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:28:45 dist-test-slave-gp6n krb5kdc[32667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:28:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:28:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:28:45.329937 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:34779
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:34779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:45.444597 32682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:45.444850 32682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:45.444891 32682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:45.449039 32682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:28:45.449095 32682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:45.449115 32682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:45.449132 32682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:28:45.449146 32682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:28:45.454324 32682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:34779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:34779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:45.455328 32682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:45.456385 32682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:45.465237 32691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:45.465183 32694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:45.465334 32682 server_base.cc:1034] running on GCE node
W20260808 18:28:45.465183 32690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:45.466032 32682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:45.467141 32682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:45.468328 32682 hybrid_clock.cc:648] HybridClock initialized: now 1786213725468308 us; error 38 us; skew 500 ppm
Aug 08 18:28:45 dist-test-slave-gp6n krb5kdc[32667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213725, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:45.471143 32682 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:28:45.472056 32682 webserver.cc:460] Webserver started at http://127.29.78.190:38413/ using document root <none> and password file <none>
I20260808 18:28:45.472577 32682 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:45.472636 32682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:45.472800 32682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:45.474864 32682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "4f742b800e0542628d614596f9b6aec4"
format_stamp: "Formatted at 2026-08-08 18:28:45 on dist-test-slave-gp6n"
server_key: "8fc3ac0eeea2f13c4c3a870769e615e2"
server_key_iv: "8ecc8faae1d25d77a5a9d86ce75a0c4b"
server_key_version: "encryptionkey@0"
I20260808 18:28:45.475363 32682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "4f742b800e0542628d614596f9b6aec4"
format_stamp: "Formatted at 2026-08-08 18:28:45 on dist-test-slave-gp6n"
server_key: "8fc3ac0eeea2f13c4c3a870769e615e2"
server_key_iv: "8ecc8faae1d25d77a5a9d86ce75a0c4b"
server_key_version: "encryptionkey@0"
I20260808 18:28:45.478950 32682 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 18:28:45.481199 32701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:45.482339 32682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:28:45.482501 32682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "4f742b800e0542628d614596f9b6aec4"
format_stamp: "Formatted at 2026-08-08 18:28:45 on dist-test-slave-gp6n"
server_key: "8fc3ac0eeea2f13c4c3a870769e615e2"
server_key_iv: "8ecc8faae1d25d77a5a9d86ce75a0c4b"
server_key_version: "encryptionkey@0"
I20260808 18:28:45.482591 32682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:45.494679 32682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:45.497427 32682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:45.497578 32682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:45.504426 32682 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:34779
I20260808 18:28:45.504452 32756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:34779 every 8 connection(s)
I20260808 18:28:45.505337 32682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:28:45.506341 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 32682
I20260808 18:28:45.506479 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:28:45.506774 30010 external_mini_cluster.cc:1428] Setting key a5e98624c488db166610ad2d43cc3fc8
I20260808 18:28:45.508632 32757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:45.513989 32757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4: Bootstrap starting.
Aug 08 18:28:45 dist-test-slave-gp6n krb5kdc[32667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:45.516471 32757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:45.517169 32757 log.cc:824] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:45.519042 32757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4: No bootstrap required, opened a new log
I20260808 18:28:45.520569 32761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:45.508192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37142 (local address 127.29.78.190:34779)
0808 18:28:45.508637 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:45.508651 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:45.508707 (+ 56us) server_negotiation.cc:407] Connection header received
0808 18:28:45.509351 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:45.509367 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:45.509683 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:45.510028 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:45.510845 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:45.511842 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:45.512707 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:45.513229 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:45.513389 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:45.516152 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:45.516157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:45.516169 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:45.516195 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:45.517872 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:45.518310 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:45.518315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:45.518319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:45.518484 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:45.518755 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:45.518758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:45.518760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:45.519149 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:45.519332 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:45.519626 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 18:28:45.519899 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":161,"threads_started":1}
I20260808 18:28:45.522881 32757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f742b800e0542628d614596f9b6aec4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34779 } }
I20260808 18:28:45.523108 32757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:45.523182 32757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f742b800e0542628d614596f9b6aec4, State: Initialized, Role: FOLLOWER
I20260808 18:28:45.523573 32757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f742b800e0542628d614596f9b6aec4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34779 } }
I20260808 18:28:45.523725 32757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:28:45.523808 32757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:28:45.523926 32757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:45.524817 32757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f742b800e0542628d614596f9b6aec4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34779 } }
I20260808 18:28:45.525152 32757 leader_election.cc:302] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f742b800e0542628d614596f9b6aec4; no voters:
I20260808 18:28:45.525453 32757 leader_election.cc:288] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:28:45.525576 32763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:45.525816 32763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4 [term 1 LEADER]: Becoming Leader. State: Replica: 4f742b800e0542628d614596f9b6aec4, State: Running, Role: LEADER
I20260808 18:28:45.526149 32763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f742b800e0542628d614596f9b6aec4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34779 } }
I20260808 18:28:45.526687 32757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:28:45.527783 32765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f742b800e0542628d614596f9b6aec4. Latest consensus state: current_term: 1 leader_uuid: "4f742b800e0542628d614596f9b6aec4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f742b800e0542628d614596f9b6aec4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34779 } } }
I20260808 18:28:45.527930 32765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4 [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:45.528318 304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:28:45.528692 32764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f742b800e0542628d614596f9b6aec4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f742b800e0542628d614596f9b6aec4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34779 } } }
I20260808 18:28:45.528843 32764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4 [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:45.531042 304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:28:45.536540 304 catalog_manager.cc:1347] Generated new cluster ID: 062125238c464c0893e93e975aff36eb
I20260808 18:28:45.536623 304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:28:45.550647 304 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:28:45.551496 304 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:28:45.564728 304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f742b800e0542628d614596f9b6aec4: Generated new TSK 0
I20260808 18:28:45.565387 304 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260808 18:28:45.605860 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34779
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:45.747303 319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:45.747565 319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:45.747608 319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:45.751555 319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:45.751611 319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:45.751667 319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:28:45.756783 319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:45.757747 319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:45.758826 319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:45.768359 325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:45.768359 327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:45.768360 329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:45.768671 319 server_base.cc:1034] running on GCE node
I20260808 18:28:45.769140 319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:45.769797 319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:45.770954 319 hybrid_clock.cc:648] HybridClock initialized: now 1786213725770937 us; error 27 us; skew 500 ppm
Aug 08 18:28:45 dist-test-slave-gp6n krb5kdc[32667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213725, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:45.773916 319 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:28:45.774783 319 webserver.cc:460] Webserver started at http://127.29.78.129:46055/ using document root <none> and password file <none>
I20260808 18:28:45.775290 319 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:45.775347 319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:45.775514 319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:45.777462 319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "185fbb5e4f74489d9d4a0221ffb8115e"
format_stamp: "Formatted at 2026-08-08 18:28:45 on dist-test-slave-gp6n"
server_key: "bbfc04f127f8464d8f8ce8451f9a2d82"
server_key_iv: "b68cd0c6ceb7ff66b547261093a6019c"
server_key_version: "encryptionkey@0"
I20260808 18:28:45.777969 319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "185fbb5e4f74489d9d4a0221ffb8115e"
format_stamp: "Formatted at 2026-08-08 18:28:45 on dist-test-slave-gp6n"
server_key: "bbfc04f127f8464d8f8ce8451f9a2d82"
server_key_iv: "b68cd0c6ceb7ff66b547261093a6019c"
server_key_version: "encryptionkey@0"
I20260808 18:28:45.782001 319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 18:28:45.784659 337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:45.785933 319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:28:45.786077 319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "185fbb5e4f74489d9d4a0221ffb8115e"
format_stamp: "Formatted at 2026-08-08 18:28:45 on dist-test-slave-gp6n"
server_key: "bbfc04f127f8464d8f8ce8451f9a2d82"
server_key_iv: "b68cd0c6ceb7ff66b547261093a6019c"
server_key_version: "encryptionkey@0"
I20260808 18:28:45.786204 319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:45.810580 319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:45.813947 319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:45.814152 319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:45.814806 319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:45.815645 319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:45.815713 319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:45.815783 319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:45.815829 319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:45.824395 319 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:39353
I20260808 18:28:45.824426 468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:39353 every 8 connection(s)
I20260808 18:28:45.825259 319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:28:45.833150 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 319
I20260808 18:28:45.833272 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:28:45.833583 30010 external_mini_cluster.cc:1428] Setting key 91d62edb0dd26c67a5a6c26f35b007a8
Aug 08 18:28:45 dist-test-slave-gp6n krb5kdc[32667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213725, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:45.839077 32761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:45.826981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:45015 (local address 127.29.78.190:34779)
0808 18:28:45.827144 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:45.827150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:45.828524 (+ 1374us) server_negotiation.cc:407] Connection header received
0808 18:28:45.829480 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:45.829484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:45.829541 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:45.829642 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:45.830831 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:45.831369 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:45.832540 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:45.832779 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:45.832887 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:45.835601 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:45.835605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:45.835607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:45.835629 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:45.837035 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:45.837531 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:45.837536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:45.837538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:45.837624 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:45.837953 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:45.837958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:45.837959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:45.838138 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:45.838244 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:45.838774 (+ 530us) server_negotiation.cc:299] Negotiation successful
0808 18:28:45.838909 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:28:45.839936 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:45.827401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34779 (local address 127.29.78.129:45015)
0808 18:28:45.828383 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:45.828426 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:45.829269 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:45.829815 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:45.829821 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:45.830166 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:45.830691 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:45.830705 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:45.831551 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:45.831554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:45.832417 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:45.832426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:45.832926 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:45.832930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:45.833099 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:45.833714 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:45.833740 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:45.835440 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:45.837149 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:45.837155 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:45.837168 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:45.837424 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:45.837713 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:45.837717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:45.837719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:45.837858 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:45.838244 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:45.838252 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:45.838550 (+ 298us) client_negotiation.cc:769] Sending connection context
0808 18:28:45.838733 (+ 183us) client_negotiation.cc:240] Negotiation successful
0808 18:28:45.839057 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":616,"threads_started":1}
I20260808 18:28:45.841105 470 heartbeater.cc:347] Connected to a master server at 127.29.78.190:34779
I20260808 18:28:45.841341 470 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:45.841822 470 heartbeater.cc:511] Master 127.29.78.190:34779 requested a full tablet report, sending...
I20260808 18:28:45.843283 32719 ts_manager.cc:194] Registered new tserver with Master: 185fbb5e4f74489d9d4a0221ffb8115e (127.29.78.129:39353)
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260808 18:28:45.844444 32719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:45015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260808 18:28:45.861562 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34779
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:45.971154 477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:45.971377 477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:45.971418 477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:45.975307 477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:45.975355 477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:45.975410 477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:28:45.980419 477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:45.981297 477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:45.982190 477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:45.991658 484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:45.991698 483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:45.991653 486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:45.992147 477 server_base.cc:1034] running on GCE node
I20260808 18:28:45.992559 477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:45.993184 477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:45.994364 477 hybrid_clock.cc:648] HybridClock initialized: now 1786213725994332 us; error 49 us; skew 500 ppm
Aug 08 18:28:45 dist-test-slave-gp6n krb5kdc[32667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213725, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:46.000494 477 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:28:46.001395 477 webserver.cc:460] Webserver started at http://127.29.78.130:46647/ using document root <none> and password file <none>
I20260808 18:28:46.001992 477 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:46.002087 477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:46.002327 477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:46.004511 477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "d38d287010b24dbf8cfea28198adf298"
format_stamp: "Formatted at 2026-08-08 18:28:46 on dist-test-slave-gp6n"
server_key: "47d2a8a38b63bf5180259cc2be147449"
server_key_iv: "4ec6fc7abb0101b935d4579b6520e8a2"
server_key_version: "encryptionkey@0"
I20260808 18:28:46.005120 477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "d38d287010b24dbf8cfea28198adf298"
format_stamp: "Formatted at 2026-08-08 18:28:46 on dist-test-slave-gp6n"
server_key: "47d2a8a38b63bf5180259cc2be147449"
server_key_iv: "4ec6fc7abb0101b935d4579b6520e8a2"
server_key_version: "encryptionkey@0"
I20260808 18:28:46.009061 477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 18:28:46.011554 493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:46.012847 477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:28:46.013090 477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "d38d287010b24dbf8cfea28198adf298"
format_stamp: "Formatted at 2026-08-08 18:28:46 on dist-test-slave-gp6n"
server_key: "47d2a8a38b63bf5180259cc2be147449"
server_key_iv: "4ec6fc7abb0101b935d4579b6520e8a2"
server_key_version: "encryptionkey@0"
I20260808 18:28:46.013198 477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:46.028564 477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:46.031893 477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:46.032087 477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:46.032665 477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:46.033505 477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:46.033571 477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:46.033639 477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:46.033686 477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:46.042974 477 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:44045
I20260808 18:28:46.043001 619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:44045 every 8 connection(s)
I20260808 18:28:46.043892 477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:28:46.047747 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 477
I20260808 18:28:46.047845 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:28:46.048137 30010 external_mini_cluster.cc:1428] Setting key 6df88289a149957baa0fb6e8943e5e63
Aug 08 18:28:46 dist-test-slave-gp6n krb5kdc[32667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213725, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:46.059309 32761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:46.045770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:33759 (local address 127.29.78.190:34779)
0808 18:28:46.045910 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:46.045914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:46.046836 (+ 922us) server_negotiation.cc:407] Connection header received
0808 18:28:46.048322 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:46.048326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:46.048396 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:46.048505 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:46.049808 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:46.050671 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:46.051925 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:46.052178 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:46.052267 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:46.055542 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:46.055545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:46.055547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:46.055570 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:46.057033 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:46.057664 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:46.057666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:46.057667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:46.057739 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:46.058142 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:46.058144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:46.058145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:46.058311 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:46.058384 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:46.059086 (+ 702us) server_negotiation.cc:299] Negotiation successful
0808 18:28:46.059186 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260808 18:28:46.060415 622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:46.046184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34779 (local address 127.29.78.130:33759)
0808 18:28:46.046688 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:46.046731 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:46.048062 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:46.048689 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:46.048698 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:46.049082 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:46.049654 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:46.049682 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:46.050791 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:46.050793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:46.051776 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:46.051790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:46.052334 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:46.052340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:46.052568 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:46.053342 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:46.053359 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:46.055344 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:46.057156 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:46.057164 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:46.057178 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:46.057565 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:46.057831 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:46.057837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:46.057840 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:46.058031 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:46.058456 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:46.058465 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:46.058808 (+ 343us) client_negotiation.cc:769] Sending connection context
0808 18:28:46.059059 (+ 251us) client_negotiation.cc:240] Negotiation successful
0808 18:28:46.059447 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":180,"threads_started":1}
I20260808 18:28:46.061612 620 heartbeater.cc:347] Connected to a master server at 127.29.78.190:34779
I20260808 18:28:46.061914 620 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:46.062489 620 heartbeater.cc:511] Master 127.29.78.190:34779 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
I20260808 18:28:46.063619 32719 ts_manager.cc:194] Registered new tserver with Master: d38d287010b24dbf8cfea28198adf298 (127.29.78.130:44045)
I20260808 18:28:46.064193 32719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:33759
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260808 18:28:46.080789 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34779
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:46.197021 625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:46.197299 625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:46.197374 625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:46.201360 625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:46.201443 625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:46.201534 625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:28:46.206722 625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:46.207751 625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:46.208801 625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:46.218447 632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:46.218428 633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:46.218578 636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:46.222380 625 server_base.cc:1034] running on GCE node
I20260808 18:28:46.222888 625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:46.223654 625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:46.224833 625 hybrid_clock.cc:648] HybridClock initialized: now 1786213726224804 us; error 43 us; skew 500 ppm
Aug 08 18:28:46 dist-test-slave-gp6n krb5kdc[32667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213726, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:46.227690 625 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:28:46.228574 625 webserver.cc:460] Webserver started at http://127.29.78.131:37167/ using document root <none> and password file <none>
I20260808 18:28:46.229122 625 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:46.229183 625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:46.229401 625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:46.231477 625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "388b96c3aa314e01a29120d4f6aba976"
format_stamp: "Formatted at 2026-08-08 18:28:46 on dist-test-slave-gp6n"
server_key: "5dc7f3aa69bf2a8c8d12aa1efd6ff61a"
server_key_iv: "b7ff2eca52ac4a50041373c8702d4311"
server_key_version: "encryptionkey@0"
I20260808 18:28:46.232067 625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "388b96c3aa314e01a29120d4f6aba976"
format_stamp: "Formatted at 2026-08-08 18:28:46 on dist-test-slave-gp6n"
server_key: "5dc7f3aa69bf2a8c8d12aa1efd6ff61a"
server_key_iv: "b7ff2eca52ac4a50041373c8702d4311"
server_key_version: "encryptionkey@0"
I20260808 18:28:46.235702 625 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:28:46.238010 644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:46.239290 625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:28:46.239418 625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "388b96c3aa314e01a29120d4f6aba976"
format_stamp: "Formatted at 2026-08-08 18:28:46 on dist-test-slave-gp6n"
server_key: "5dc7f3aa69bf2a8c8d12aa1efd6ff61a"
server_key_iv: "b7ff2eca52ac4a50041373c8702d4311"
server_key_version: "encryptionkey@0"
I20260808 18:28:46.239518 625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:46.252786 625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:46.255568 625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:46.255753 625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:46.256323 625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:46.257081 625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:46.257128 625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:46.257164 625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:46.257216 625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:46.265852 625 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:44231
I20260808 18:28:46.265879 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:44231 every 8 connection(s)
I20260808 18:28:46.266741 625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:28:46.267469 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 625
I20260808 18:28:46.267729 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:28:46.268162 30010 external_mini_cluster.cc:1428] Setting key 77edd980439500a6a7388034d745dc30
Aug 08 18:28:46 dist-test-slave-gp6n krb5kdc[32667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213726, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:46.280927 32761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:46.268490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:55383 (local address 127.29.78.190:34779)
0808 18:28:46.268606 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:46.268610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:46.269652 (+ 1042us) server_negotiation.cc:407] Connection header received
0808 18:28:46.271085 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:46.271089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:46.271143 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:46.271220 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:46.272329 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:46.272869 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:46.274381 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:46.274682 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:46.274797 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:46.277257 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:46.277261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:46.277264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:46.277300 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:46.278742 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:46.279292 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:46.279298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:46.279301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:46.279398 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:46.279768 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:46.279773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:46.279775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:46.279975 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:46.280088 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:46.280585 (+ 497us) server_negotiation.cc:299] Negotiation successful
0808 18:28:46.280729 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 18:28:46.281983 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:46.268901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34779 (local address 127.29.78.131:55383)
0808 18:28:46.269508 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:46.269565 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:46.270889 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:46.271386 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:46.271393 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:46.271732 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:46.272182 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:46.272195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:46.273186 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:46.273190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:46.274234 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:46.274244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:46.274865 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:46.274870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:46.275043 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:46.275583 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:46.275599 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:46.277089 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:46.278871 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:46.278877 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:46.278891 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:46.279180 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:46.279506 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:46.279509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:46.279512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:46.279666 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:46.280087 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:46.280094 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:46.280333 (+ 239us) client_negotiation.cc:769] Sending connection context
0808 18:28:46.280529 (+ 196us) client_negotiation.cc:240] Negotiation successful
0808 18:28:46.280878 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":128,"threads_started":1}
I20260808 18:28:46.283295 769 heartbeater.cc:347] Connected to a master server at 127.29.78.190:34779
I20260808 18:28:46.283581 769 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:46.284192 769 heartbeater.cc:511] Master 127.29.78.190:34779 requested a full tablet report, sending...
I20260808 18:28:46.285558 32719 ts_manager.cc:194] Registered new tserver with Master: 388b96c3aa314e01a29120d4f6aba976 (127.29.78.131:44231)
I20260808 18:28:46.286502 32719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:55383
I20260808 18:28:46.293887 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 18:28:46.309417 32761 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:28:46.307950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37146 (local address 127.29.78.190:34779)
0808 18:28:46.308077 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:46.308080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:46.308265 (+ 185us) server_negotiation.cc:407] Connection header received
0808 18:28:46.308388 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:46.308391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:46.308435 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:46.308517 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:46.309289 (+ 772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37146: BlockingRecv error: recv got EOF from 127.0.0.1:37146 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 18:28:46.310773 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 319
I20260808 18:28:46.319305 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 477
I20260808 18:28:46.326100 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 625
I20260808 18:28:46.332495 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 32682
2026-08-08T18:28:46Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3143 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:28:46 dist-test-slave-gp6n krb5kdc[782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:28:46 dist-test-slave-gp6n krb5kdc[782](info): set up 2 sockets
Aug 08 18:28:46 dist-test-slave-gp6n krb5kdc[782](info): commencing operation
krb5kdc: starting...
W20260808 18:28:47.434171 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:28:47 dist-test-slave-gp6n krb5kdc[782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:28:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:28:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:28:47.562453 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:37217
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:43257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:37217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:47.671840 797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:47.672124 797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:47.672192 797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:47.676146 797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:28:47.676229 797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:47.676266 797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:47.676306 797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:28:47.676342 797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:28:47.681413 797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:43257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:37217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:37217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:47.682415 797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:47.683398 797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:28:47.691627 797 server_base.cc:1034] running on GCE node
W20260808 18:28:47.691578 807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:47.691570 803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:47.691551 805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:47.692392 797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:47.693431 797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:47.694584 797 hybrid_clock.cc:648] HybridClock initialized: now 1786213727694566 us; error 29 us; skew 500 ppm
Aug 08 18:28:47 dist-test-slave-gp6n krb5kdc[782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213727, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:47.697190 797 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:28:47.698048 797 webserver.cc:460] Webserver started at http://127.29.78.190:33165/ using document root <none> and password file <none>
I20260808 18:28:47.698634 797 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:47.698720 797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:47.698932 797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:47.700856 797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "38a1d0861026442eb673bb5334389299"
format_stamp: "Formatted at 2026-08-08 18:28:47 on dist-test-slave-gp6n"
server_key: "569c1d49b4d671c05479a7982d6c56b3"
server_key_iv: "ce00bcfb2b0e93b34554418cdee6b1cf"
server_key_version: "encryptionkey@0"
I20260808 18:28:47.701426 797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "38a1d0861026442eb673bb5334389299"
format_stamp: "Formatted at 2026-08-08 18:28:47 on dist-test-slave-gp6n"
server_key: "569c1d49b4d671c05479a7982d6c56b3"
server_key_iv: "ce00bcfb2b0e93b34554418cdee6b1cf"
server_key_version: "encryptionkey@0"
I20260808 18:28:47.705410 797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 18:28:47.708010 815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:47.709182 797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:28:47.709355 797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "38a1d0861026442eb673bb5334389299"
format_stamp: "Formatted at 2026-08-08 18:28:47 on dist-test-slave-gp6n"
server_key: "569c1d49b4d671c05479a7982d6c56b3"
server_key_iv: "ce00bcfb2b0e93b34554418cdee6b1cf"
server_key_version: "encryptionkey@0"
I20260808 18:28:47.709518 797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:47.725212 797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:47.728389 797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:47.728636 797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:47.736186 797 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:37217
I20260808 18:28:47.736644 870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:37217 every 8 connection(s)
I20260808 18:28:47.737238 797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:28:47.738566 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 797
I20260808 18:28:47.738718 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:28:47.739076 30010 external_mini_cluster.cc:1428] Setting key 7cb637639efc5bea7e538db207467c99
I20260808 18:28:47.740937 871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:47.748123 871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299: Bootstrap starting.
Aug 08 18:28:47 dist-test-slave-gp6n krb5kdc[782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:47.750998 871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:47.751987 871 log.cc:824] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:47.754429 871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299: No bootstrap required, opened a new log
I20260808 18:28:47.755332 874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:47.740507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37982 (local address 127.29.78.190:37217)
0808 18:28:47.740977 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:47.740994 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:47.741050 (+ 56us) server_negotiation.cc:407] Connection header received
0808 18:28:47.741828 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:47.741846 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:47.742195 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:47.742580 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:47.743543 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:47.744786 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:47.746090 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:47.746663 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:47.746790 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:47.749833 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:47.749840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:47.749856 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:47.749889 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:47.752065 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:47.752615 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:47.752620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:47.752625 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:47.752741 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:47.753047 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:47.753050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:47.753052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:47.753557 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:47.753739 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:47.754023 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 18:28:47.754346 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":155,"threads_started":1}
I20260808 18:28:47.758479 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:28:47.758828 871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a1d0861026442eb673bb5334389299" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37217 } }
I20260808 18:28:47.759123 871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:47.759205 871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a1d0861026442eb673bb5334389299, State: Initialized, Role: FOLLOWER
I20260808 18:28:47.759683 871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a1d0861026442eb673bb5334389299" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37217 } }
I20260808 18:28:47.759865 871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:28:47.759950 871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:28:47.760057 871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:47.761389 871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a1d0861026442eb673bb5334389299" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37217 } }
I20260808 18:28:47.761725 871 leader_election.cc:302] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38a1d0861026442eb673bb5334389299; no voters:
I20260808 18:28:47.762022 871 leader_election.cc:288] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:28:47.762189 877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:47.762585 877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299 [term 1 LEADER]: Becoming Leader. State: Replica: 38a1d0861026442eb673bb5334389299, State: Running, Role: LEADER
I20260808 18:28:47.763104 877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a1d0861026442eb673bb5334389299" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37217 } }
I20260808 18:28:47.763283 871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:28:47.764806 879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38a1d0861026442eb673bb5334389299. Latest consensus state: current_term: 1 leader_uuid: "38a1d0861026442eb673bb5334389299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a1d0861026442eb673bb5334389299" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37217 } } }
I20260808 18:28:47.764950 879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299 [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:47.765261 878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38a1d0861026442eb673bb5334389299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a1d0861026442eb673bb5334389299" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37217 } } }
I20260808 18:28:47.765336 878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299 [sys.catalog]: This master's current role is: LEADER
W20260808 18:28:47.768119 894 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:28:47.768205 894 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:28:47.768316 891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:28:47.768996 891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:28:47.775642 891 catalog_manager.cc:1347] Generated new cluster ID: fa620e88c9074c3fa9617018947b7dee
I20260808 18:28:47.775713 891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:28:47.780539 874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:47.774660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37996 (local address 127.29.78.190:37217)
0808 18:28:47.775282 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:47.775287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:47.775303 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:28:47.775344 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:47.775347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:47.775392 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:47.775489 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:47.776136 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:47.776931 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:47.778159 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:47.778532 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:47.778625 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:47.779585 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:47.779589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:47.779592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:28:47.780093 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:47.780123 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:28:47.780141 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:47.780220 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:47.780351 (+ 131us) server_negotiation.cc:299] Negotiation successful
0808 18:28:47.780409 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542}
I20260808 18:28:47.785279 891 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:28:47.786155 891 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:28:47.795029 891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38a1d0861026442eb673bb5334389299: Generated new TSK 0
I20260808 18:28:47.795698 891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:28:47.802618 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 797
2026-08-08T18:28:47Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1471 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:28:47 dist-test-slave-gp6n krb5kdc[907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:28:47 dist-test-slave-gp6n krb5kdc[907](info): set up 2 sockets
Aug 08 18:28:47 dist-test-slave-gp6n krb5kdc[907](info): commencing operation
krb5kdc: starting...
W20260808 18:28:48.968580 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:28:49 dist-test-slave-gp6n krb5kdc[907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:28:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:28:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:28:49.097712 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:43307
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:43307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:49.208021 923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:49.208329 923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:49.208411 923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:49.212399 923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:28:49.212479 923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:49.212519 923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:49.212574 923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:28:49.212615 923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:28:49.218720 923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:43307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:43307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:49.220232 923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:49.221693 923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:49.232519 930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:49.232532 933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:49.232529 931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:49.233016 923 server_base.cc:1034] running on GCE node
I20260808 18:28:49.233574 923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:49.234803 923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:49.235996 923 hybrid_clock.cc:648] HybridClock initialized: now 1786213729235961 us; error 55 us; skew 500 ppm
Aug 08 18:28:49 dist-test-slave-gp6n krb5kdc[907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213729, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:49.239729 923 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:28:49.241015 923 webserver.cc:460] Webserver started at http://127.29.78.190:42737/ using document root <none> and password file <none>
I20260808 18:28:49.241788 923 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:49.241914 923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:49.242185 923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:49.245350 923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "a195bdf6dc4048c0856e32386e050d44"
format_stamp: "Formatted at 2026-08-08 18:28:49 on dist-test-slave-gp6n"
server_key: "271c0e323b482c098cc77d0dfc322039"
server_key_iv: "02149a9ee7a99f0e967de321425d0af4"
server_key_version: "encryptionkey@0"
I20260808 18:28:49.246090 923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "a195bdf6dc4048c0856e32386e050d44"
format_stamp: "Formatted at 2026-08-08 18:28:49 on dist-test-slave-gp6n"
server_key: "271c0e323b482c098cc77d0dfc322039"
server_key_iv: "02149a9ee7a99f0e967de321425d0af4"
server_key_version: "encryptionkey@0"
I20260808 18:28:49.250265 923 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260808 18:28:49.252612 942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:49.253991 923 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:28:49.254096 923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "a195bdf6dc4048c0856e32386e050d44"
format_stamp: "Formatted at 2026-08-08 18:28:49 on dist-test-slave-gp6n"
server_key: "271c0e323b482c098cc77d0dfc322039"
server_key_iv: "02149a9ee7a99f0e967de321425d0af4"
server_key_version: "encryptionkey@0"
I20260808 18:28:49.254179 923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:49.270231 923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:49.273280 923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:49.273447 923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:49.280690 923 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:43307
I20260808 18:28:49.280978 999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:43307 every 8 connection(s)
I20260808 18:28:49.281729 923 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:28:49.284209 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 923
I20260808 18:28:49.284344 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:28:49.284638 1000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:49.284657 30010 external_mini_cluster.cc:1428] Setting key 0d36241811620623a6ed5727d6180a13
I20260808 18:28:49.292053 1000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44: Bootstrap starting.
I20260808 18:28:49.294654 1000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44: Neither blocks nor log segments found. Creating new log.
Aug 08 18:28:49 dist-test-slave-gp6n krb5kdc[907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:49.295646 1000 log.cc:824] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:49.298166 1000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44: No bootstrap required, opened a new log
I20260808 18:28:49.301374 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:49.286044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43344 (local address 127.29.78.190:43307)
0808 18:28:49.286480 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:49.286499 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:49.286541 (+ 42us) server_negotiation.cc:407] Connection header received
0808 18:28:49.287521 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:49.287545 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:49.287975 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:49.288346 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:49.289081 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:49.290635 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:49.291525 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:49.292213 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:49.292369 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:49.295847 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:49.295854 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:49.295871 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:49.295914 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:49.298073 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:49.298676 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:49.298682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:49.298688 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:49.298829 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:49.299108 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:49.299111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:49.299112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:49.299598 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:49.299794 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:49.300023 (+ 229us) server_negotiation.cc:299] Negotiation successful
0808 18:28:49.300333 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":107,"threads_started":1}
I20260808 18:28:49.302771 1000 raft_consensus.cc:348] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a195bdf6dc4048c0856e32386e050d44" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43307 } }
I20260808 18:28:49.302999 1000 raft_consensus.cc:374] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:49.303035 1000 raft_consensus.cc:729] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a195bdf6dc4048c0856e32386e050d44, State: Initialized, Role: FOLLOWER
I20260808 18:28:49.303529 1000 consensus_queue.cc:260] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a195bdf6dc4048c0856e32386e050d44" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43307 } }
I20260808 18:28:49.303695 1000 raft_consensus.cc:388] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:28:49.303761 1000 raft_consensus.cc:482] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:28:49.303864 1000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:49.304737 1000 raft_consensus.cc:504] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a195bdf6dc4048c0856e32386e050d44" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43307 } }
I20260808 18:28:49.305061 1000 leader_election.cc:302] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a195bdf6dc4048c0856e32386e050d44; no voters:
I20260808 18:28:49.305346 1000 leader_election.cc:288] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:28:49.305534 1006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:49.305789 1006 raft_consensus.cc:686] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44 [term 1 LEADER]: Becoming Leader. State: Replica: a195bdf6dc4048c0856e32386e050d44, State: Running, Role: LEADER
I20260808 18:28:49.306103 1006 consensus_queue.cc:237] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a195bdf6dc4048c0856e32386e050d44" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43307 } }
I20260808 18:28:49.306663 1000 sys_catalog.cc:564] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:28:49.307958 1007 sys_catalog.cc:455] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a195bdf6dc4048c0856e32386e050d44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a195bdf6dc4048c0856e32386e050d44" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43307 } } }
I20260808 18:28:49.307981 1008 sys_catalog.cc:455] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a195bdf6dc4048c0856e32386e050d44. Latest consensus state: current_term: 1 leader_uuid: "a195bdf6dc4048c0856e32386e050d44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a195bdf6dc4048c0856e32386e050d44" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43307 } } }
I20260808 18:28:49.308112 1008 sys_catalog.cc:458] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44 [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:49.308112 1007 sys_catalog.cc:458] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44 [sys.catalog]: This master's current role is: LEADER
W20260808 18:28:49.311659 1025 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:28:49.311749 1025 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:28:49.311854 1020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:28:49.312757 1020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:28:49.318281 1020 catalog_manager.cc:1347] Generated new cluster ID: e246b92fac2949be8206a90748057347
I20260808 18:28:49.318351 1020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:28:49.345774 1020 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:28:49.346783 1020 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:28:49.356565 1020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a195bdf6dc4048c0856e32386e050d44: Generated new TSK 0
I20260808 18:28:49.357281 1020 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260808 18:28:49.393388 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43307
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:49.509388 1033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:49.509627 1033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:49.509668 1033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:49.513518 1033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:49.513569 1033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:49.513624 1033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:28:49.518621 1033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:49.519619 1033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:49.520552 1033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:49.530267 1041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:49.530264 1039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:49.530260 1038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:49.530668 1033 server_base.cc:1034] running on GCE node
I20260808 18:28:49.531095 1033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:49.531711 1033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:49.532895 1033 hybrid_clock.cc:648] HybridClock initialized: now 1786213729532872 us; error 35 us; skew 500 ppm
Aug 08 18:28:49 dist-test-slave-gp6n krb5kdc[907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213729, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:49.535709 1033 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:28:49.536520 1033 webserver.cc:460] Webserver started at http://127.29.78.129:40609/ using document root <none> and password file <none>
I20260808 18:28:49.537061 1033 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:49.537148 1033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:49.537374 1033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:49.539444 1033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "a1c868ffba6f4fd392f8cf0d19a74c12"
format_stamp: "Formatted at 2026-08-08 18:28:49 on dist-test-slave-gp6n"
server_key: "d4e8c623eb910ab89b393f786c15858b"
server_key_iv: "76c0a8c3cc819a894882c869bb626100"
server_key_version: "encryptionkey@0"
I20260808 18:28:49.540022 1033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "a1c868ffba6f4fd392f8cf0d19a74c12"
format_stamp: "Formatted at 2026-08-08 18:28:49 on dist-test-slave-gp6n"
server_key: "d4e8c623eb910ab89b393f786c15858b"
server_key_iv: "76c0a8c3cc819a894882c869bb626100"
server_key_version: "encryptionkey@0"
I20260808 18:28:49.543776 1033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:28:49.546074 1048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:49.547227 1033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:28:49.547355 1033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "a1c868ffba6f4fd392f8cf0d19a74c12"
format_stamp: "Formatted at 2026-08-08 18:28:49 on dist-test-slave-gp6n"
server_key: "d4e8c623eb910ab89b393f786c15858b"
server_key_iv: "76c0a8c3cc819a894882c869bb626100"
server_key_version: "encryptionkey@0"
I20260808 18:28:49.547466 1033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:49.574199 1033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:49.577113 1033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:49.577302 1033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:49.577878 1033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:49.578697 1033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:49.578763 1033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:49.578832 1033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:49.578872 1033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:49.587693 1033 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:43899
I20260808 18:28:49.587730 1175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:43899 every 8 connection(s)
I20260808 18:28:49.588531 1033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:28:49.590181 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 1033
I20260808 18:28:49.590301 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:28:49.590621 30010 external_mini_cluster.cc:1428] Setting key fec2ec09c1bb2092b1131552463fafa1
Aug 08 18:28:49 dist-test-slave-gp6n krb5kdc[907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213729, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260808 18:28:49.604198 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:49.591838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:50327 (local address 127.29.78.190:43307)
0808 18:28:49.591991 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:49.591995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:49.592011 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:28:49.593076 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:49.593082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:49.593138 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:49.593226 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:49.594300 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:49.595175 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:49.597038 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:49.597278 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:49.597370 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:49.600349 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:49.600354 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:49.600369 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:49.600404 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:49.602046 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:49.602656 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:49.602659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:49.602660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:49.602736 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:49.603092 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:49.603095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:49.603097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:49.603277 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:49.603397 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:49.603874 (+ 477us) server_negotiation.cc:299] Negotiation successful
0808 18:28:49.604003 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 18:28:49.605103 1178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:49.590836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43307 (local address 127.29.78.129:50327)
0808 18:28:49.591328 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:49.591385 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:49.592879 (+ 1494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:49.593353 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:49.593360 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:49.593693 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:49.594157 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:49.594171 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:49.595376 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:49.595382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:49.596901 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:49.596916 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:49.597433 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:49.597439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:49.597700 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:49.598302 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:49.598341 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:49.600143 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:49.602168 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:49.602174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:49.602188 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:49.602541 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:49.602827 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:49.602831 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:49.602833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:49.602988 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:49.603385 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:49.603391 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:49.603635 (+ 244us) client_negotiation.cc:769] Sending connection context
0808 18:28:49.603812 (+ 177us) client_negotiation.cc:240] Negotiation successful
0808 18:28:49.604144 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":137,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
I20260808 18:28:49.606474 1176 heartbeater.cc:347] Connected to a master server at 127.29.78.190:43307
I20260808 18:28:49.606714 1176 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:49.607262 1176 heartbeater.cc:511] Master 127.29.78.190:43307 requested a full tablet report, sending...
I20260808 18:28:49.608927 961 ts_manager.cc:194] Registered new tserver with Master: a1c868ffba6f4fd392f8cf0d19a74c12 (127.29.78.129:43899)
I20260808 18:28:49.610142 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:50327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260808 18:28:49.621618 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43307
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:49.731285 1181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:49.731554 1181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:49.731621 1181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:49.735589 1181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:49.735662 1181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:49.735741 1181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:28:49.740828 1181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:49.741815 1181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:49.742847 1181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:49.752254 1190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:49.752238 1186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:49.752241 1188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:49.752775 1181 server_base.cc:1034] running on GCE node
I20260808 18:28:49.753211 1181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:49.753813 1181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:49.755002 1181 hybrid_clock.cc:648] HybridClock initialized: now 1786213729754973 us; error 50 us; skew 500 ppm
Aug 08 18:28:49 dist-test-slave-gp6n krb5kdc[907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213729, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:49.757849 1181 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:28:49.758819 1181 webserver.cc:460] Webserver started at http://127.29.78.130:44437/ using document root <none> and password file <none>
I20260808 18:28:49.759354 1181 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:49.759438 1181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:49.759649 1181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:49.761619 1181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "bc3a58fe46184782a383e4aeb6cf1256"
format_stamp: "Formatted at 2026-08-08 18:28:49 on dist-test-slave-gp6n"
server_key: "a5e9e2a6bc3ef2f6f204f8ab6977b503"
server_key_iv: "84a9b6668c218ace6b2288e9ba560d62"
server_key_version: "encryptionkey@0"
I20260808 18:28:49.762183 1181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "bc3a58fe46184782a383e4aeb6cf1256"
format_stamp: "Formatted at 2026-08-08 18:28:49 on dist-test-slave-gp6n"
server_key: "a5e9e2a6bc3ef2f6f204f8ab6977b503"
server_key_iv: "84a9b6668c218ace6b2288e9ba560d62"
server_key_version: "encryptionkey@0"
I20260808 18:28:49.765786 1181 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:28:49.768215 1197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:49.769490 1181 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 18:28:49.769634 1181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "bc3a58fe46184782a383e4aeb6cf1256"
format_stamp: "Formatted at 2026-08-08 18:28:49 on dist-test-slave-gp6n"
server_key: "a5e9e2a6bc3ef2f6f204f8ab6977b503"
server_key_iv: "84a9b6668c218ace6b2288e9ba560d62"
server_key_version: "encryptionkey@0"
I20260808 18:28:49.769747 1181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:49.796090 1181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:49.799242 1181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:49.799448 1181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:49.800027 1181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:49.800837 1181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:49.800905 1181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:49.800976 1181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:49.801016 1181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:49.809878 1181 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:36295
I20260808 18:28:49.809918 1323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:36295 every 8 connection(s)
I20260808 18:28:49.810807 1181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:28:49.818202 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 1181
I20260808 18:28:49.818336 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:28:49.818683 30010 external_mini_cluster.cc:1428] Setting key 8fc3c88c9614d8dcd82ed281435d9f29
Aug 08 18:28:49 dist-test-slave-gp6n krb5kdc[907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213729, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:49.825026 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:49.812593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:40693 (local address 127.29.78.190:43307)
0808 18:28:49.812745 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:49.812750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:49.813576 (+ 826us) server_negotiation.cc:407] Connection header received
0808 18:28:49.814981 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:49.814985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:49.815041 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:49.815123 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:49.816277 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:49.816861 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:49.818094 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:49.818390 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:49.818517 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:49.821163 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:49.821170 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:49.821177 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:49.821211 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:49.822806 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:49.823484 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:49.823487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:49.823489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:49.823579 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:49.823956 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:49.823961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:49.823963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:49.824161 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:49.824270 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:49.824739 (+ 469us) server_negotiation.cc:299] Negotiation successful
0808 18:28:49.824865 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 18:28:49.825932 1326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:49.812985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43307 (local address 127.29.78.130:40693)
0808 18:28:49.813431 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:49.813474 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:49.814779 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:49.815309 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:49.815317 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:49.815668 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:49.816135 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:49.816149 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:49.817001 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:49.817004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:49.817950 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:49.817965 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:49.818571 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:49.818575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:49.818773 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:49.819446 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:49.819468 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:49.820999 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:49.822931 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:49.822939 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:49.822957 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:49.823341 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:49.823681 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:49.823683 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:49.823685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:49.823848 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:49.824267 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:49.824274 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:49.824510 (+ 236us) client_negotiation.cc:769] Sending connection context
0808 18:28:49.824696 (+ 186us) client_negotiation.cc:240] Negotiation successful
0808 18:28:49.825029 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":115,"threads_started":1}
I20260808 18:28:49.827132 1324 heartbeater.cc:347] Connected to a master server at 127.29.78.190:43307
I20260808 18:28:49.827370 1324 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:49.827865 1324 heartbeater.cc:511] Master 127.29.78.190:43307 requested a full tablet report, sending...
I20260808 18:28:49.828971 961 ts_manager.cc:194] Registered new tserver with Master: bc3a58fe46184782a383e4aeb6cf1256 (127.29.78.130:36295)
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260808 18:28:49.829773 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:40693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260808 18:28:49.850056 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43307
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:49.960656 1329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:49.960888 1329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:49.960928 1329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:49.964854 1329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:49.964906 1329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:49.964996 1329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:28:49.970074 1329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:49.971087 1329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:49.972079 1329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:49.981659 1335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:49.981727 1329 server_base.cc:1034] running on GCE node
W20260808 18:28:49.981616 1337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:49.981616 1334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:49.982317 1329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:49.982995 1329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:49.984196 1329 hybrid_clock.cc:648] HybridClock initialized: now 1786213729984179 us; error 26 us; skew 500 ppm
Aug 08 18:28:49 dist-test-slave-gp6n krb5kdc[907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213729, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:49.987030 1329 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:28:49.987911 1329 webserver.cc:460] Webserver started at http://127.29.78.131:39445/ using document root <none> and password file <none>
I20260808 18:28:49.988463 1329 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:49.988549 1329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:49.988768 1329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:49.990832 1329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "0162cd496096401789d10af5c87162e8"
format_stamp: "Formatted at 2026-08-08 18:28:49 on dist-test-slave-gp6n"
server_key: "2d99992f574508d72116b0c4164c0e40"
server_key_iv: "438bccb750e1d96155874eaf437f1e4a"
server_key_version: "encryptionkey@0"
I20260808 18:28:49.991374 1329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "0162cd496096401789d10af5c87162e8"
format_stamp: "Formatted at 2026-08-08 18:28:49 on dist-test-slave-gp6n"
server_key: "2d99992f574508d72116b0c4164c0e40"
server_key_iv: "438bccb750e1d96155874eaf437f1e4a"
server_key_version: "encryptionkey@0"
I20260808 18:28:49.995076 1329 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:28:49.997453 1344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:49.998725 1329 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:28:49.998845 1329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "0162cd496096401789d10af5c87162e8"
format_stamp: "Formatted at 2026-08-08 18:28:49 on dist-test-slave-gp6n"
server_key: "2d99992f574508d72116b0c4164c0e40"
server_key_iv: "438bccb750e1d96155874eaf437f1e4a"
server_key_version: "encryptionkey@0"
I20260808 18:28:49.998950 1329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:50.015698 1329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:50.018877 1329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:50.019081 1329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:50.019702 1329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:50.020527 1329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:50.020594 1329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:50.020661 1329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:50.020706 1329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:50.029796 1329 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:33669
I20260808 18:28:50.029841 1474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:33669 every 8 connection(s)
I20260808 18:28:50.030748 1329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:28:50.036917 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 1329
I20260808 18:28:50.037052 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:28:50.037349 30010 external_mini_cluster.cc:1428] Setting key 07b3b3057d6f22fd0b3c9aee3c66246a
Aug 08 18:28:50 dist-test-slave-gp6n krb5kdc[907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213729, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:50.044183 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:50.032496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:42307 (local address 127.29.78.190:43307)
0808 18:28:50.032660 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:50.032665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:50.033450 (+ 785us) server_negotiation.cc:407] Connection header received
0808 18:28:50.034841 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:50.034845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:50.034893 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:50.034975 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:50.036083 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:50.036607 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:50.037892 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:50.038187 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:50.038295 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:50.040876 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:50.040879 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:50.040881 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:50.040904 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:50.042259 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:50.042802 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:50.042806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:50.042807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:50.042905 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:50.043228 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:50.043231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:50.043232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:50.043421 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:50.043505 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:50.043947 (+ 442us) server_negotiation.cc:299] Negotiation successful
0808 18:28:50.044055 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 18:28:50.044977 1477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:50.032869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43307 (local address 127.29.78.131:42307)
0808 18:28:50.033296 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:50.033338 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:50.034641 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:50.035129 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:50.035135 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:50.035488 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:50.035957 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:50.035969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:50.036733 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:50.036736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:50.037761 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:50.037770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:50.038332 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:50.038337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:50.038541 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:50.039143 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:50.039159 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:50.040689 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:50.042370 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:50.042374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:50.042392 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:50.042701 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:50.043000 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:50.043002 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:50.043003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:50.043136 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:50.043511 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:50.043516 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:50.043731 (+ 215us) client_negotiation.cc:769] Sending connection context
0808 18:28:50.043876 (+ 145us) client_negotiation.cc:240] Negotiation successful
0808 18:28:50.044116 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":107,"threads_started":1}
I20260808 18:28:50.046196 1475 heartbeater.cc:347] Connected to a master server at 127.29.78.190:43307
I20260808 18:28:50.046492 1475 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:50.046985 1475 heartbeater.cc:511] Master 127.29.78.190:43307 requested a full tablet report, sending...
I20260808 18:28:50.047948 961 ts_manager.cc:194] Registered new tserver with Master: 0162cd496096401789d10af5c87162e8 (127.29.78.131:33669)
I20260808 18:28:50.048516 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:42307
I20260808 18:28:50.051144 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 18:28:50 dist-test-slave-gp6n krb5kdc[907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213730, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 08 18:28:50 dist-test-slave-gp6n krb5kdc[907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213730, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:50.077612 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:50.069068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43360 (local address 127.29.78.190:43307)
0808 18:28:50.069199 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:50.069203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:50.069413 (+ 210us) server_negotiation.cc:407] Connection header received
0808 18:28:50.069532 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:50.069535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:50.069576 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:50.069651 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:50.070320 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:50.070888 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:50.071713 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:50.071923 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:50.072012 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:50.074810 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:50.074813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:50.074815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:50.074838 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:50.076118 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:50.076502 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:50.076504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:50.076505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:50.076576 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:50.076854 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:50.076856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:50.076857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:50.077072 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:50.077173 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:50.077395 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 18:28:50.077494 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260808 18:28:50.077987 961 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43360
I20260808 18:28:50.079010 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 1033
I20260808 18:28:50.088891 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 1181
I20260808 18:28:50.095944 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 1329
I20260808 18:28:50.104159 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 923
2026-08-08T18:28:50Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2297 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:28:50 dist-test-slave-gp6n krb5kdc[1487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:28:50 dist-test-slave-gp6n krb5kdc[1487](info): set up 2 sockets
Aug 08 18:28:50 dist-test-slave-gp6n krb5kdc[1487](info): commencing operation
krb5kdc: starting...
W20260808 18:28:51.198907 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:28:51 dist-test-slave-gp6n krb5kdc[1487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:28:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:28:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:28:51.333508 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:36393
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:36393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:51.447248 1501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:51.447587 1501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:51.447666 1501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:51.451753 1501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:28:51.451845 1501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:51.451892 1501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:51.451946 1501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:28:51.451984 1501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:28:51.457733 1501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:36393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:36393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:51.458915 1501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:51.459972 1501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:28:51.468732 1501 server_base.cc:1034] running on GCE node
W20260808 18:28:51.468673 1507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:51.468679 1508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:51.468673 1510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:51.469617 1501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:51.470856 1501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:51.472071 1501 hybrid_clock.cc:648] HybridClock initialized: now 1786213731472040 us; error 40 us; skew 500 ppm
Aug 08 18:28:51 dist-test-slave-gp6n krb5kdc[1487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213731, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:51.481323 1501 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:28:51.482285 1501 webserver.cc:460] Webserver started at http://127.29.78.190:44175/ using document root <none> and password file <none>
I20260808 18:28:51.482900 1501 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:51.482998 1501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:51.483233 1501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:51.485332 1501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "847535ade67343b68f9405bce4d35319"
format_stamp: "Formatted at 2026-08-08 18:28:51 on dist-test-slave-gp6n"
server_key: "fd270cbde5ae4583c13ddf37aadcd2e6"
server_key_iv: "2b6724ca151a47d1c4bca67abbdb5426"
server_key_version: "encryptionkey@0"
I20260808 18:28:51.485936 1501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "847535ade67343b68f9405bce4d35319"
format_stamp: "Formatted at 2026-08-08 18:28:51 on dist-test-slave-gp6n"
server_key: "fd270cbde5ae4583c13ddf37aadcd2e6"
server_key_iv: "2b6724ca151a47d1c4bca67abbdb5426"
server_key_version: "encryptionkey@0"
I20260808 18:28:51.490159 1501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260808 18:28:51.492812 1519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:51.494117 1501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:28:51.494274 1501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "847535ade67343b68f9405bce4d35319"
format_stamp: "Formatted at 2026-08-08 18:28:51 on dist-test-slave-gp6n"
server_key: "fd270cbde5ae4583c13ddf37aadcd2e6"
server_key_iv: "2b6724ca151a47d1c4bca67abbdb5426"
server_key_version: "encryptionkey@0"
I20260808 18:28:51.494392 1501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:51.503901 1501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:51.506902 1501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:51.507090 1501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:51.513810 1501 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:36393
I20260808 18:28:51.513921 1578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:36393 every 8 connection(s)
I20260808 18:28:51.514820 1501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:28:51.517578 1579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:51.520742 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 1501
I20260808 18:28:51.520852 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:28:51.521160 30010 external_mini_cluster.cc:1428] Setting key d70d2697cf846fa9eb17f51d80f6f8cc
I20260808 18:28:51.523639 1579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319: Bootstrap starting.
I20260808 18:28:51.526296 1579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:51.527192 1579 log.cc:824] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:51.529446 1579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319: No bootstrap required, opened a new log
Aug 08 18:28:51 dist-test-slave-gp6n krb5kdc[1487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:51.532894 1579 raft_consensus.cc:348] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847535ade67343b68f9405bce4d35319" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36393 } }
I20260808 18:28:51.533113 1579 raft_consensus.cc:374] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:51.533146 1579 raft_consensus.cc:729] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 847535ade67343b68f9405bce4d35319, State: Initialized, Role: FOLLOWER
I20260808 18:28:51.533560 1579 consensus_queue.cc:260] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847535ade67343b68f9405bce4d35319" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36393 } }
I20260808 18:28:51.533680 1579 raft_consensus.cc:388] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:28:51.533769 1579 raft_consensus.cc:482] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:28:51.533867 1579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:51.534947 1579 raft_consensus.cc:504] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847535ade67343b68f9405bce4d35319" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36393 } }
I20260808 18:28:51.535331 1579 leader_election.cc:302] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 847535ade67343b68f9405bce4d35319; no voters:
I20260808 18:28:51.535666 1579 leader_election.cc:288] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:28:51.535765 1584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:51.536007 1584 raft_consensus.cc:686] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319 [term 1 LEADER]: Becoming Leader. State: Replica: 847535ade67343b68f9405bce4d35319, State: Running, Role: LEADER
I20260808 18:28:51.536497 1584 consensus_queue.cc:237] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847535ade67343b68f9405bce4d35319" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36393 } }
I20260808 18:28:51.536976 1579 sys_catalog.cc:564] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:28:51.537696 1584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 847535ade67343b68f9405bce4d35319. Latest consensus state: current_term: 1 leader_uuid: "847535ade67343b68f9405bce4d35319" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847535ade67343b68f9405bce4d35319" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36393 } } }
I20260808 18:28:51.537845 1584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319 [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:51.537878 1582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:51.522586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44502 (local address 127.29.78.190:36393)
0808 18:28:51.523002 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:51.523020 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:51.523061 (+ 41us) server_negotiation.cc:407] Connection header received
0808 18:28:51.523886 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:51.523908 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:51.524303 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:51.524650 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:51.525379 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:51.526941 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:51.528123 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:51.528784 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:51.528948 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:51.531705 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:51.531710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:51.531723 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:51.531748 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:51.534269 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:51.534792 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:51.534796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:51.534801 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:51.534911 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:51.535213 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:51.535216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:51.535217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:51.535691 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:51.536543 (+ 852us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:51.536698 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 18:28:51.536942 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":109,"threads_started":1}
I20260808 18:28:51.538355 1585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "847535ade67343b68f9405bce4d35319" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847535ade67343b68f9405bce4d35319" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36393 } } }
I20260808 18:28:51.538563 1585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319 [sys.catalog]: This master's current role is: LEADER
W20260808 18:28:51.542574 1601 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:28:51.542639 1601 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:28:51.542742 1593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:28:51.543650 1593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:28:51.549204 1593 catalog_manager.cc:1347] Generated new cluster ID: 9249f2b387b64881a8decbdef3548d22
I20260808 18:28:51.549273 1593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:28:51.564713 1593 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:28:51.565914 1593 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:28:51.572757 1593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 847535ade67343b68f9405bce4d35319: Generated new TSK 0
I20260808 18:28:51.573523 1593 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260808 18:28:51.603470 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36393
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:51.715406 1607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:51.715668 1607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:51.715734 1607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:51.719681 1607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:51.719754 1607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:51.719835 1607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:28:51.724927 1607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:51.725929 1607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:51.726991 1607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:51.736182 1613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:51.736178 1616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:51.736227 1614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:51.736563 1607 server_base.cc:1034] running on GCE node
I20260808 18:28:51.736964 1607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:51.737651 1607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:51.738818 1607 hybrid_clock.cc:648] HybridClock initialized: now 1786213731738805 us; error 32 us; skew 500 ppm
Aug 08 18:28:51 dist-test-slave-gp6n krb5kdc[1487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213731, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:51.741660 1607 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:28:51.742566 1607 webserver.cc:460] Webserver started at http://127.29.78.129:41871/ using document root <none> and password file <none>
I20260808 18:28:51.743101 1607 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:51.743186 1607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:51.743409 1607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:51.745384 1607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "8890df5ecef44886ae508d34674049de"
format_stamp: "Formatted at 2026-08-08 18:28:51 on dist-test-slave-gp6n"
server_key: "5c0ae9f1620d80e875755dbda2f44aa2"
server_key_iv: "36ecb2c87ff6368c30fe905a044a0c71"
server_key_version: "encryptionkey@0"
I20260808 18:28:51.745950 1607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "8890df5ecef44886ae508d34674049de"
format_stamp: "Formatted at 2026-08-08 18:28:51 on dist-test-slave-gp6n"
server_key: "5c0ae9f1620d80e875755dbda2f44aa2"
server_key_iv: "36ecb2c87ff6368c30fe905a044a0c71"
server_key_version: "encryptionkey@0"
I20260808 18:28:51.749624 1607 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 18:28:51.752175 1625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:51.753582 1607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:28:51.753708 1607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "8890df5ecef44886ae508d34674049de"
format_stamp: "Formatted at 2026-08-08 18:28:51 on dist-test-slave-gp6n"
server_key: "5c0ae9f1620d80e875755dbda2f44aa2"
server_key_iv: "36ecb2c87ff6368c30fe905a044a0c71"
server_key_version: "encryptionkey@0"
I20260808 18:28:51.753819 1607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:51.775604 1607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:51.778864 1607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:51.779081 1607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:51.779690 1607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:51.780531 1607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:51.780596 1607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:51.780666 1607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:51.780715 1607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:51.789911 1607 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:41603
I20260808 18:28:51.790031 1755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:41603 every 8 connection(s)
I20260808 18:28:51.790858 1607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
Aug 08 18:28:51 dist-test-slave-gp6n krb5kdc[1487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213731, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:51.800367 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 1607
I20260808 18:28:51.800501 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:28:51.800807 30010 external_mini_cluster.cc:1428] Setting key 7620c3db4827aac25f5f779788de6088
I20260808 18:28:51.805056 1582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:51.792610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:39513 (local address 127.29.78.190:36393)
0808 18:28:51.792760 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:51.792765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:51.793672 (+ 907us) server_negotiation.cc:407] Connection header received
0808 18:28:51.795051 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:51.795054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:51.795102 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:51.795182 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:51.796266 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:51.796805 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:51.797948 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:51.798167 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:51.798246 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:51.800992 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:51.800996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:51.800999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:51.801033 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:51.802836 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:51.803397 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:51.803403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:51.803405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:51.803498 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:51.803876 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:51.803881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:51.803883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:51.804081 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:51.804193 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:51.804722 (+ 529us) server_negotiation.cc:299] Negotiation successful
0808 18:28:51.804867 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 18:28:51.805917 1758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:51.793028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36393 (local address 127.29.78.129:39513)
0808 18:28:51.793481 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:51.793525 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:51.794855 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:51.795307 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:51.795314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:51.795662 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:51.796131 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:51.796145 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:51.796906 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:51.796908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:51.797836 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:51.797850 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:51.798292 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:51.798294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:51.798463 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:51.799075 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:51.799091 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:51.800791 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:51.802962 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:51.802968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:51.802983 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:51.803275 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:51.803609 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:51.803612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:51.803614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:51.803769 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:51.804205 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:51.804214 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:51.804475 (+ 261us) client_negotiation.cc:769] Sending connection context
0808 18:28:51.804676 (+ 201us) client_negotiation.cc:240] Negotiation successful
0808 18:28:51.805024 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":120,"threads_started":1}
I20260808 18:28:51.807323 1756 heartbeater.cc:347] Connected to a master server at 127.29.78.190:36393
I20260808 18:28:51.807574 1756 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:51.808043 1756 heartbeater.cc:511] Master 127.29.78.190:36393 requested a full tablet report, sending...
I20260808 18:28:51.809691 1538 ts_manager.cc:194] Registered new tserver with Master: 8890df5ecef44886ae508d34674049de (127.29.78.129:41603)
I20260808 18:28:51.810885 1538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:39513
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260808 18:28:51.829653 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36393
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:51.944394 1761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:51.944713 1761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:51.944803 1761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:51.948906 1761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:51.948988 1761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:51.949076 1761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:28:51.954558 1761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:51.955646 1761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:51.956700 1761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:51.966279 1767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:51.966274 1770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:51.966293 1766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:51.966519 1761 server_base.cc:1034] running on GCE node
I20260808 18:28:51.966960 1761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:51.967613 1761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:51.968793 1761 hybrid_clock.cc:648] HybridClock initialized: now 1786213731968773 us; error 33 us; skew 500 ppm
Aug 08 18:28:51 dist-test-slave-gp6n krb5kdc[1487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213731, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:51.971565 1761 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:28:51.972407 1761 webserver.cc:460] Webserver started at http://127.29.78.130:39853/ using document root <none> and password file <none>
I20260808 18:28:51.972906 1761 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:51.972963 1761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:51.973134 1761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:51.975188 1761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "9d3ba453ce434910801c4bc6e1dbb6ba"
format_stamp: "Formatted at 2026-08-08 18:28:51 on dist-test-slave-gp6n"
server_key: "618d7cde39d20ff2525389107e2e7262"
server_key_iv: "20db024da2b81575c4ca1a5dec5bbb48"
server_key_version: "encryptionkey@0"
I20260808 18:28:51.975703 1761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "9d3ba453ce434910801c4bc6e1dbb6ba"
format_stamp: "Formatted at 2026-08-08 18:28:51 on dist-test-slave-gp6n"
server_key: "618d7cde39d20ff2525389107e2e7262"
server_key_iv: "20db024da2b81575c4ca1a5dec5bbb48"
server_key_version: "encryptionkey@0"
I20260808 18:28:51.979367 1761 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:28:51.981695 1778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:51.982976 1761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:28:51.983098 1761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "9d3ba453ce434910801c4bc6e1dbb6ba"
format_stamp: "Formatted at 2026-08-08 18:28:51 on dist-test-slave-gp6n"
server_key: "618d7cde39d20ff2525389107e2e7262"
server_key_iv: "20db024da2b81575c4ca1a5dec5bbb48"
server_key_version: "encryptionkey@0"
I20260808 18:28:51.983201 1761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:52.011190 1761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:52.014092 1761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:52.014290 1761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:52.014920 1761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:52.015728 1761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:52.015799 1761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:52.015878 1761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:52.015923 1761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:52.024511 1761 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:38555
I20260808 18:28:52.024554 1906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:38555 every 8 connection(s)
I20260808 18:28:52.025372 1761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:28:52.026183 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 1761
I20260808 18:28:52.026319 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:28:52.026672 30010 external_mini_cluster.cc:1428] Setting key 4ba756f413f825d87879a33a54045848
Aug 08 18:28:52 dist-test-slave-gp6n krb5kdc[1487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213731, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
I20260808 18:28:52.041503 1582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:52.027346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:44225 (local address 127.29.78.190:36393)
0808 18:28:52.027502 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:52.027508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:52.028409 (+ 901us) server_negotiation.cc:407] Connection header received
0808 18:28:52.029834 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:52.029847 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:52.029912 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:52.030008 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:52.031775 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:52.032683 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:52.033854 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:52.034323 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:52.034511 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:52.037515 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:52.037518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:52.037520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:52.037587 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:52.039374 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:52.039899 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:52.039907 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:52.039908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:52.039992 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:52.040335 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:52.040338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:52.040340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:52.040558 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:52.040697 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:52.041195 (+ 498us) server_negotiation.cc:299] Negotiation successful
0808 18:28:52.041328 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 18:28:52.042793 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:52.027758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36393 (local address 127.29.78.130:44225)
0808 18:28:52.028242 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:52.028296 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:52.029627 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:52.030235 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:52.030247 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:52.030874 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:52.031586 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:52.031609 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:52.032829 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:52.032832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:52.033732 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:52.033741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:52.034551 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:52.034554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:52.034698 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:52.035381 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:52.035402 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:52.037329 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:52.039495 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:52.039501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:52.039517 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:52.039785 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:52.040088 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:52.040092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:52.040094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:52.040232 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:52.040663 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:52.040669 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:52.040939 (+ 270us) client_negotiation.cc:769] Sending connection context
0808 18:28:52.041135 (+ 196us) client_negotiation.cc:240] Negotiation successful
0808 18:28:52.041482 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":132,"threads_started":1}
I20260808 18:28:52.044080 1907 heartbeater.cc:347] Connected to a master server at 127.29.78.190:36393
I20260808 18:28:52.044332 1907 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:52.044838 1907 heartbeater.cc:511] Master 127.29.78.190:36393 requested a full tablet report, sending...
I20260808 18:28:52.046056 1538 ts_manager.cc:194] Registered new tserver with Master: 9d3ba453ce434910801c4bc6e1dbb6ba (127.29.78.130:38555)
I20260808 18:28:52.046981 1538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:44225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260808 18:28:52.059346 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36393
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:52.170300 1912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:52.170648 1912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:52.170693 1912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:52.174693 1912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:52.174750 1912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:52.174808 1912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:28:52.180050 1912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:52.181121 1912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:52.182166 1912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:52.191834 1920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:52.191793 1922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:52.191793 1919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:52.192129 1912 server_base.cc:1034] running on GCE node
I20260808 18:28:52.192571 1912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:52.193181 1912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:52.194340 1912 hybrid_clock.cc:648] HybridClock initialized: now 1786213732194310 us; error 39 us; skew 500 ppm
Aug 08 18:28:52 dist-test-slave-gp6n krb5kdc[1487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213732, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:52.197212 1912 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:28:52.198104 1912 webserver.cc:460] Webserver started at http://127.29.78.131:38457/ using document root <none> and password file <none>
I20260808 18:28:52.198678 1912 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:52.198768 1912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:52.198964 1912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:52.200898 1912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "4ff067a3a599478283064611a425f392"
format_stamp: "Formatted at 2026-08-08 18:28:52 on dist-test-slave-gp6n"
server_key: "7c8edadd832751e3633cd36dbcb0eae5"
server_key_iv: "4f65a60e6bb39bb7534cfbcc6d16228d"
server_key_version: "encryptionkey@0"
I20260808 18:28:52.201416 1912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "4ff067a3a599478283064611a425f392"
format_stamp: "Formatted at 2026-08-08 18:28:52 on dist-test-slave-gp6n"
server_key: "7c8edadd832751e3633cd36dbcb0eae5"
server_key_iv: "4f65a60e6bb39bb7534cfbcc6d16228d"
server_key_version: "encryptionkey@0"
I20260808 18:28:52.205135 1912 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 18:28:52.207620 1930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:52.208841 1912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:28:52.208984 1912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "4ff067a3a599478283064611a425f392"
format_stamp: "Formatted at 2026-08-08 18:28:52 on dist-test-slave-gp6n"
server_key: "7c8edadd832751e3633cd36dbcb0eae5"
server_key_iv: "4f65a60e6bb39bb7534cfbcc6d16228d"
server_key_version: "encryptionkey@0"
I20260808 18:28:52.209101 1912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:52.243163 1912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:52.246919 1912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:52.247156 1912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:52.247800 1912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:52.248672 1912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:52.248742 1912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:52.248811 1912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:52.248860 1912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:52.258354 1912 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:40417
I20260808 18:28:52.258464 2061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:40417 every 8 connection(s)
I20260808 18:28:52.259274 1912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:28:52.266417 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 1912
I20260808 18:28:52.266546 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:28:52.266891 30010 external_mini_cluster.cc:1428] Setting key 56a4f0f7a90d7bc94916f947969ac0cf
Aug 08 18:28:52 dist-test-slave-gp6n krb5kdc[1487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213732, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:52.273231 1582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:52.261152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:36793 (local address 127.29.78.190:36393)
0808 18:28:52.261307 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:52.261310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:52.262112 (+ 802us) server_negotiation.cc:407] Connection header received
0808 18:28:52.263553 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:52.263556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:52.263605 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:52.263683 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:52.264814 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:52.265345 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:52.266856 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:52.267113 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:52.267196 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:52.269809 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:52.269813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:52.269816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:52.269840 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:52.271418 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:52.271853 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:52.271856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:52.271857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:52.271963 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:52.272250 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:52.272252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:52.272254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:52.272473 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:52.272554 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:52.272994 (+ 440us) server_negotiation.cc:299] Negotiation successful
0808 18:28:52.273116 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 18:28:52.273933 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:52.261507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36393 (local address 127.29.78.131:36793)
0808 18:28:52.261945 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:52.261990 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:52.263335 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:52.263835 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:52.263841 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:52.264183 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:52.264663 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:52.264676 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:52.265484 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:52.265487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:52.266733 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:52.266752 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:52.267261 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:52.267264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:52.267456 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:52.268046 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:52.268070 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:52.269638 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:52.271511 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:52.271515 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:52.271526 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:52.271763 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:52.272040 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:52.272042 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:52.272044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:52.272165 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:52.272560 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:52.272564 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:52.272763 (+ 199us) client_negotiation.cc:769] Sending connection context
0808 18:28:52.272909 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 18:28:52.273139 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":117,"threads_started":1}
I20260808 18:28:52.275139 2062 heartbeater.cc:347] Connected to a master server at 127.29.78.190:36393
I20260808 18:28:52.275331 2062 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:52.275743 2062 heartbeater.cc:511] Master 127.29.78.190:36393 requested a full tablet report, sending...
I20260808 18:28:52.276738 1538 ts_manager.cc:194] Registered new tserver with Master: 4ff067a3a599478283064611a425f392 (127.29.78.131:40417)
I20260808 18:28:52.277349 1538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:36793
I20260808 18:28:52.280721 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 18:28:52 dist-test-slave-gp6n krb5kdc[1487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 18:28:52 dist-test-slave-gp6n krb5kdc[1487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260808 18:28:52.309721 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:52.299131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34802 (local address 127.29.78.129:41603)
0808 18:28:52.299460 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:52.299482 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:52.299509 (+ 27us) server_negotiation.cc:407] Connection header received
0808 18:28:52.299569 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:52.299574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:52.299762 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:52.299892 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:52.300545 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:52.301326 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:52.302131 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:52.302591 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:52.302681 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:52.305050 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:52.305059 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:52.305072 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:52.305096 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:52.307187 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:52.307790 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:52.307798 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:52.307802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:52.307924 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:52.308243 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:52.308248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:52.308250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:52.308564 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:52.308787 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:52.309195 (+ 408us) server_negotiation.cc:299] Negotiation successful
0808 18:28:52.309580 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":86,"threads_started":1}
Aug 08 18:28:52 dist-test-slave-gp6n krb5kdc[1487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:52.321141 1582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:52.312796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44516 (local address 127.29.78.190:36393)
0808 18:28:52.312953 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:52.312958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:52.313064 (+ 106us) server_negotiation.cc:407] Connection header received
0808 18:28:52.313158 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:52.313161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:52.313212 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:52.313275 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:52.313965 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:52.314599 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:52.315405 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:52.315613 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:52.315696 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:52.318181 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:52.318184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:52.318186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:52.318208 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:52.319598 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:52.320020 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:52.320023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:52.320025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:52.320118 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:52.320377 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:52.320379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:52.320381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:52.320609 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:52.320727 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:52.320915 (+ 188us) server_negotiation.cc:299] Negotiation successful
0808 18:28:52.321017 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260808 18:28:52.321578 1538 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44516
I20260808 18:28:52.322271 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 1607
I20260808 18:28:52.330845 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 1761
I20260808 18:28:52.339305 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 1912
I20260808 18:28:52.347747 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 1501
2026-08-08T18:28:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2244 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:28:52 dist-test-slave-gp6n krb5kdc[2077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:28:52 dist-test-slave-gp6n krb5kdc[2077](info): set up 2 sockets
Aug 08 18:28:52 dist-test-slave-gp6n krb5kdc[2077](info): commencing operation
krb5kdc: starting...
W20260808 18:28:53.434712 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:28:53 dist-test-slave-gp6n krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:28:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:28:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:28:53.567752 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:41097
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:34795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:41097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:53.684015 2090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:53.684269 2090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:53.684314 2090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:53.688282 2090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:28:53.688329 2090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:53.688349 2090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:53.688364 2090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:28:53.688406 2090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:28:53.693548 2090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:34795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:41097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:41097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:53.694584 2090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:53.695606 2090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:53.703848 2098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:53.703877 2096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:53.703927 2090 server_base.cc:1034] running on GCE node
W20260808 18:28:53.703890 2095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:53.704608 2090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:53.705648 2090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:53.706830 2090 hybrid_clock.cc:648] HybridClock initialized: now 1786213733706819 us; error 45 us; skew 500 ppm
Aug 08 18:28:53 dist-test-slave-gp6n krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213733, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:53.709573 2090 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:28:53.710265 2090 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:53.710356 2090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:53.710598 2090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:53.712618 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "b2260a2ec58f4661990ead6020705595"
format_stamp: "Formatted at 2026-08-08 18:28:53 on dist-test-slave-gp6n"
server_key: "213c58aac7b03798f7357e210eb1ffa3"
server_key_iv: "a98ded5b3af34c9b296f834b36377905"
server_key_version: "encryptionkey@0"
I20260808 18:28:53.713191 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "b2260a2ec58f4661990ead6020705595"
format_stamp: "Formatted at 2026-08-08 18:28:53 on dist-test-slave-gp6n"
server_key: "213c58aac7b03798f7357e210eb1ffa3"
server_key_iv: "a98ded5b3af34c9b296f834b36377905"
server_key_version: "encryptionkey@0"
I20260808 18:28:53.716826 2090 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 18:28:53.719187 2106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:53.720440 2090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:28:53.720572 2090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "b2260a2ec58f4661990ead6020705595"
format_stamp: "Formatted at 2026-08-08 18:28:53 on dist-test-slave-gp6n"
server_key: "213c58aac7b03798f7357e210eb1ffa3"
server_key_iv: "a98ded5b3af34c9b296f834b36377905"
server_key_version: "encryptionkey@0"
I20260808 18:28:53.720676 2090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:53.734196 2090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:53.737066 2090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:53.737264 2090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:53.744184 2090 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:41097
I20260808 18:28:53.744417 2162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:41097 every 8 connection(s)
I20260808 18:28:53.744884 2090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:28:53.747666 2163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:53.753268 2163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595: Bootstrap starting.
I20260808 18:28:53.754519 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 2090
I20260808 18:28:53.754663 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:28:53.754935 30010 external_mini_cluster.cc:1428] Setting key 0b167280ed9a1db2dd1f540b249bd589
I20260808 18:28:53.756075 2163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:53.756856 2163 log.cc:824] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:53.759145 2163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595: No bootstrap required, opened a new log
I20260808 18:28:53.763367 2163 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2260a2ec58f4661990ead6020705595" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41097 } }
I20260808 18:28:53.763660 2163 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:53.763732 2163 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2260a2ec58f4661990ead6020705595, State: Initialized, Role: FOLLOWER
I20260808 18:28:53.764163 2163 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2260a2ec58f4661990ead6020705595" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41097 } }
I20260808 18:28:53.764314 2163 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:28:53.764396 2163 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Aug 08 18:28:53 dist-test-slave-gp6n krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:53.764508 2163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:53.765642 2163 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2260a2ec58f4661990ead6020705595" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41097 } }
I20260808 18:28:53.766011 2163 leader_election.cc:302] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2260a2ec58f4661990ead6020705595; no voters:
I20260808 18:28:53.766433 2163 leader_election.cc:288] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:28:53.766510 2169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:53.766721 2169 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595 [term 1 LEADER]: Becoming Leader. State: Replica: b2260a2ec58f4661990ead6020705595, State: Running, Role: LEADER
I20260808 18:28:53.767059 2169 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2260a2ec58f4661990ead6020705595" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41097 } }
I20260808 18:28:53.767709 2163 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:28:53.768675 2170 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2260a2ec58f4661990ead6020705595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2260a2ec58f4661990ead6020705595" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41097 } } }
I20260808 18:28:53.768837 2170 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595 [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:53.769181 2171 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2260a2ec58f4661990ead6020705595. Latest consensus state: current_term: 1 leader_uuid: "b2260a2ec58f4661990ead6020705595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2260a2ec58f4661990ead6020705595" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41097 } } }
I20260808 18:28:53.769258 2171 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595 [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:53.770126 2178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:28:53.771212 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:53.756022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60096 (local address 127.29.78.190:41097)
0808 18:28:53.756445 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:53.756472 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:53.756509 (+ 37us) server_negotiation.cc:407] Connection header received
0808 18:28:53.757279 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:53.757316 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:53.757709 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:53.758016 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:53.758989 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:53.760143 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:53.761352 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:53.761837 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:53.761970 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:53.765134 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:53.765139 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:53.765152 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:53.765178 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:53.767332 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:53.767764 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:53.767771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:53.767778 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:53.767926 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:53.768235 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:53.768238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:53.768240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:53.768753 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:53.768934 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:53.769906 (+ 972us) server_negotiation.cc:299] Negotiation successful
0808 18:28:53.770239 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":156,"threads_started":1}
I20260808 18:28:53.773262 2178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:28:53.778782 2178 catalog_manager.cc:1347] Generated new cluster ID: 0fecee5a80524fb8955964e2cf9c9ccc
I20260808 18:28:53.778895 2178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:28:53.785737 2178 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:28:53.786659 2178 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:28:53.798389 2178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595: Generated new TSK 0
I20260808 18:28:53.799229 2178 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260808 18:28:53.827901 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41097
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:34795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:53.941046 2192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:53.941303 2192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:53.941346 2192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:53.945420 2192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:53.945477 2192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:53.945569 2192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:28:53.950780 2192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:34795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:53.951848 2192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:53.952888 2192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:53.962540 2198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:53.962566 2199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:53.962540 2201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:53.962608 2192 server_base.cc:1034] running on GCE node
I20260808 18:28:53.963186 2192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:53.963840 2192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:53.965045 2192 hybrid_clock.cc:648] HybridClock initialized: now 1786213733965020 us; error 43 us; skew 500 ppm
Aug 08 18:28:53 dist-test-slave-gp6n krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213733, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:53.968132 2192 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:28:53.968911 2192 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:53.969000 2192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:53.969228 2192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:53.971454 2192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "cf5002a4e76b4519b186c7a392ce4016"
format_stamp: "Formatted at 2026-08-08 18:28:53 on dist-test-slave-gp6n"
server_key: "0d211c9c058d870e54ea8fb54e752698"
server_key_iv: "d9199cf4d5b993d6a3a8452480a85e2f"
server_key_version: "encryptionkey@0"
I20260808 18:28:53.972031 2192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "cf5002a4e76b4519b186c7a392ce4016"
format_stamp: "Formatted at 2026-08-08 18:28:53 on dist-test-slave-gp6n"
server_key: "0d211c9c058d870e54ea8fb54e752698"
server_key_iv: "d9199cf4d5b993d6a3a8452480a85e2f"
server_key_version: "encryptionkey@0"
I20260808 18:28:53.975713 2192 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:28:53.978001 2210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:53.979288 2192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:28:53.979485 2192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "cf5002a4e76b4519b186c7a392ce4016"
format_stamp: "Formatted at 2026-08-08 18:28:53 on dist-test-slave-gp6n"
server_key: "0d211c9c058d870e54ea8fb54e752698"
server_key_iv: "d9199cf4d5b993d6a3a8452480a85e2f"
server_key_version: "encryptionkey@0"
I20260808 18:28:53.979604 2192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:53.993824 2192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:53.996757 2192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:53.996975 2192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:53.997555 2192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:53.998449 2192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:53.998534 2192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:53.998617 2192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:53.998668 2192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:54.008256 2192 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:35535
I20260808 18:28:54.008287 2334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:35535 every 8 connection(s)
I20260808 18:28:54.008875 2192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:28:54.014671 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 2192
I20260808 18:28:54.014796 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:28:54.015118 30010 external_mini_cluster.cc:1428] Setting key 270b36b62fa7ad247ec0a59f645f0cb2
Aug 08 18:28:54 dist-test-slave-gp6n krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213733, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:54.023514 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:54.010734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:45833 (local address 127.29.78.190:41097)
0808 18:28:54.010874 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:54.010878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:54.011703 (+ 825us) server_negotiation.cc:407] Connection header received
0808 18:28:54.012992 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:54.012994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:54.013042 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:54.013119 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:54.014271 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.015005 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.016442 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.016744 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.016845 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:54.019785 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:54.019788 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:54.019790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:54.019815 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:54.021318 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:54.021847 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:54.021851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:54.021853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:54.021945 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:54.022282 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:54.022287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:54.022288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:54.022545 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:54.022672 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:54.023174 (+ 502us) server_negotiation.cc:299] Negotiation successful
0808 18:28:54.023325 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 18:28:54.024387 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:54.011090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:41097 (local address 127.29.78.129:45833)
0808 18:28:54.011549 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:54.011590 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:54.012790 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:54.013277 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:54.013283 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:54.013658 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:54.014126 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:54.014140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.015149 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.015155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:54.016278 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:54.016288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.016922 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.016929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:54.017156 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:54.017788 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:54.017814 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:54.019610 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:54.021461 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:54.021465 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:54.021478 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:54.021734 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:54.022037 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:54.022040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:54.022042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:54.022185 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:54.022672 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:54.022680 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:54.022928 (+ 248us) client_negotiation.cc:769] Sending connection context
0808 18:28:54.023115 (+ 187us) client_negotiation.cc:240] Negotiation successful
0808 18:28:54.023454 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":122,"threads_started":1}
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260808 18:28:54.025628 2336 heartbeater.cc:347] Connected to a master server at 127.29.78.190:41097
I20260808 18:28:54.025868 2336 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:54.026350 2336 heartbeater.cc:511] Master 127.29.78.190:41097 requested a full tablet report, sending...
I20260808 18:28:54.027973 2126 ts_manager.cc:194] Registered new tserver with Master: cf5002a4e76b4519b186c7a392ce4016 (127.29.78.129:35535)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
I20260808 18:28:54.029160 2126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:45833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260808 18:28:54.044962 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41097
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:34795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:54.157122 2342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:54.157363 2342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:54.157405 2342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:54.161664 2342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:54.161720 2342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:54.161809 2342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:28:54.166904 2342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:34795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:54.167914 2342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:54.168998 2342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:54.178330 2352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:54.178319 2350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:54.178300 2349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:54.178318 2342 server_base.cc:1034] running on GCE node
I20260808 18:28:54.178958 2342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:54.179638 2342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:54.180831 2342 hybrid_clock.cc:648] HybridClock initialized: now 1786213734180816 us; error 32 us; skew 500 ppm
Aug 08 18:28:54 dist-test-slave-gp6n krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213734, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:54.183969 2342 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:28:54.184681 2342 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:54.184746 2342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:54.184955 2342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:54.187052 2342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "657e59ae44ea44ffbdd3015755118d55"
format_stamp: "Formatted at 2026-08-08 18:28:54 on dist-test-slave-gp6n"
server_key: "c849d40cde218ee413b7a2c0e1dee3b0"
server_key_iv: "925b43ddf1d504e114e789d3bdf5cd1c"
server_key_version: "encryptionkey@0"
I20260808 18:28:54.187652 2342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "657e59ae44ea44ffbdd3015755118d55"
format_stamp: "Formatted at 2026-08-08 18:28:54 on dist-test-slave-gp6n"
server_key: "c849d40cde218ee413b7a2c0e1dee3b0"
server_key_iv: "925b43ddf1d504e114e789d3bdf5cd1c"
server_key_version: "encryptionkey@0"
I20260808 18:28:54.191413 2342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:28:54.193756 2359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:54.195091 2342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:28:54.195227 2342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "657e59ae44ea44ffbdd3015755118d55"
format_stamp: "Formatted at 2026-08-08 18:28:54 on dist-test-slave-gp6n"
server_key: "c849d40cde218ee413b7a2c0e1dee3b0"
server_key_iv: "925b43ddf1d504e114e789d3bdf5cd1c"
server_key_version: "encryptionkey@0"
I20260808 18:28:54.195333 2342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:54.224504 2342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:54.227734 2342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:54.227891 2342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:54.228418 2342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:54.229254 2342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:54.229336 2342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:54.229404 2342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:54.229450 2342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:54.238256 2342 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:33785
I20260808 18:28:54.238288 2487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:33785 every 8 connection(s)
I20260808 18:28:54.238867 2342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:28:54.241979 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 2342
I20260808 18:28:54.242139 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:28:54.242551 30010 external_mini_cluster.cc:1428] Setting key e263fe26f40ba4ce399d88eacbf4c99a
Aug 08 18:28:54 dist-test-slave-gp6n krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213734, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260808 18:28:54.254933 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:54.240670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:48073 (local address 127.29.78.190:41097)
0808 18:28:54.240815 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:54.240821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:54.241706 (+ 885us) server_negotiation.cc:407] Connection header received
0808 18:28:54.243630 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:54.243633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:54.243678 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:54.243757 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:54.245469 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.246017 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.247511 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.247905 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.248042 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:54.250870 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:54.250874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:54.250876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:54.250904 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:54.252486 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:54.253171 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:54.253173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:54.253180 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:54.253265 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:54.253690 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:54.253692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:54.253694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:54.253877 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:54.254000 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:54.254683 (+ 683us) server_negotiation.cc:299] Negotiation successful
0808 18:28:54.254797 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 18:28:54.255874 2490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:54.241101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:41097 (local address 127.29.78.130:48073)
0808 18:28:54.241536 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:54.241578 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:54.243349 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:54.243997 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:54.244006 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:54.244532 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:54.245287 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:54.245312 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.246145 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.246152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:54.247372 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:54.247382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.248084 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.248087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:54.248242 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:54.248963 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:54.248979 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:54.250681 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:54.252610 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:54.252618 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:54.252636 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:54.253013 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:54.253380 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:54.253386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:54.253389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:54.253584 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:54.254020 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:54.254028 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:54.254337 (+ 309us) client_negotiation.cc:769] Sending connection context
0808 18:28:54.254580 (+ 243us) client_negotiation.cc:240] Negotiation successful
0808 18:28:54.254837 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
I20260808 18:28:54.257087 2488 heartbeater.cc:347] Connected to a master server at 127.29.78.190:41097
I20260808 18:28:54.257367 2488 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:54.257967 2488 heartbeater.cc:511] Master 127.29.78.190:41097 requested a full tablet report, sending...
I20260808 18:28:54.259114 2126 ts_manager.cc:194] Registered new tserver with Master: 657e59ae44ea44ffbdd3015755118d55 (127.29.78.130:33785)
I20260808 18:28:54.259693 2126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:48073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260808 18:28:54.274318 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41097
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:34795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:54.389269 2493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:54.389550 2493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:54.389599 2493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:54.393626 2493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:54.393692 2493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:54.393779 2493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:28:54.399051 2493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:34795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:54.400112 2493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:54.401119 2493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:54.410861 2503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:54.410900 2500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:54.411077 2493 server_base.cc:1034] running on GCE node
W20260808 18:28:54.411197 2501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:54.411655 2493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:54.412259 2493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:54.413416 2493 hybrid_clock.cc:648] HybridClock initialized: now 1786213734413395 us; error 34 us; skew 500 ppm
Aug 08 18:28:54 dist-test-slave-gp6n krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213734, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:54.416368 2493 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:28:54.417181 2493 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:54.417281 2493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:54.417514 2493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:54.419601 2493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "08aec525c786476fb11c6a2a5376f113"
format_stamp: "Formatted at 2026-08-08 18:28:54 on dist-test-slave-gp6n"
server_key: "d63348952f688ed6679b9f77723abecc"
server_key_iv: "574e78fd553ebb48747bae3007e2db90"
server_key_version: "encryptionkey@0"
I20260808 18:28:54.420159 2493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "08aec525c786476fb11c6a2a5376f113"
format_stamp: "Formatted at 2026-08-08 18:28:54 on dist-test-slave-gp6n"
server_key: "d63348952f688ed6679b9f77723abecc"
server_key_iv: "574e78fd553ebb48747bae3007e2db90"
server_key_version: "encryptionkey@0"
I20260808 18:28:54.423882 2493 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:28:54.426270 2510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:54.427595 2493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:28:54.427726 2493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "08aec525c786476fb11c6a2a5376f113"
format_stamp: "Formatted at 2026-08-08 18:28:54 on dist-test-slave-gp6n"
server_key: "d63348952f688ed6679b9f77723abecc"
server_key_iv: "574e78fd553ebb48747bae3007e2db90"
server_key_version: "encryptionkey@0"
I20260808 18:28:54.427834 2493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:54.443157 2493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:54.446377 2493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:54.446631 2493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:54.447216 2493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:54.448056 2493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:54.448122 2493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:54.448194 2493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:54.448235 2493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:54.457011 2493 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:37807
I20260808 18:28:54.457033 2636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:37807 every 8 connection(s)
I20260808 18:28:54.457621 2493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:28:54.461470 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 2493
I20260808 18:28:54.461637 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:28:54.462052 30010 external_mini_cluster.cc:1428] Setting key fc1962bf0542a4fc4db1b55d581094e6
Aug 08 18:28:54 dist-test-slave-gp6n krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213734, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:54.471652 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:54.459469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:47785 (local address 127.29.78.190:41097)
0808 18:28:54.459608 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:54.459612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:54.460432 (+ 820us) server_negotiation.cc:407] Connection header received
0808 18:28:54.461755 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:54.461759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:54.461824 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:54.461936 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:54.463287 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.463834 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.465016 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.465239 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.465349 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:54.467950 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:54.467954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:54.467956 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:54.467982 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:54.469519 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:54.470060 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:54.470064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:54.470065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:54.470151 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:54.470545 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:54.470549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:54.470550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:54.470742 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:54.470847 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:54.471354 (+ 507us) server_negotiation.cc:299] Negotiation successful
0808 18:28:54.471481 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 18:28:54.472506 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:54.459831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:41097 (local address 127.29.78.131:47785)
0808 18:28:54.460262 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:54.460304 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:54.461509 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:54.462111 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:54.462119 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:54.462649 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:54.463146 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:54.463161 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.463968 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.463977 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:54.464865 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:54.464874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.465386 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.465390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:54.465561 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:54.466161 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:54.466178 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:54.467775 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:54.469638 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:54.469644 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:54.469679 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:54.469958 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:54.470243 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:54.470246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:54.470248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:54.470390 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:54.470846 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:54.470852 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:54.471108 (+ 256us) client_negotiation.cc:769] Sending connection context
0808 18:28:54.471290 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 18:28:54.471633 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
I20260808 18:28:54.473629 2638 heartbeater.cc:347] Connected to a master server at 127.29.78.190:41097
I20260808 18:28:54.473891 2638 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:54.474467 2638 heartbeater.cc:511] Master 127.29.78.190:41097 requested a full tablet report, sending...
I20260808 18:28:54.475675 2126 ts_manager.cc:194] Registered new tserver with Master: 08aec525c786476fb11c6a2a5376f113 (127.29.78.131:37807)
I20260808 18:28:54.476274 2126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:47785
I20260808 18:28:54.487807 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:28:54.497320 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:54.489991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60104 (local address 127.29.78.190:41097)
0808 18:28:54.490136 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:54.490141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:54.490288 (+ 147us) server_negotiation.cc:407] Connection header received
0808 18:28:54.490392 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:54.490394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:54.490521 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:54.490593 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:54.491277 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.491818 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.492609 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.492813 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.492888 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:54.494206 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:54.494209 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:54.494211 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:54.494234 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:54.495796 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:54.496190 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:54.496192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:54.496194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:54.496268 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:54.496493 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:54.496496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:54.496497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:54.496709 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:54.496811 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:54.497013 (+ 202us) server_negotiation.cc:299] Negotiation successful
0808 18:28:54.497107 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 18:28:54.500252 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:28:54.503278 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:28:54.517800 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:54.511502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:37807 (local address 127.0.0.1:46978)
0808 18:28:54.512326 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:54.512377 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:54.512497 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:54.513414 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:54.513422 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:54.513452 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:54.513620 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:54.513631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.515166 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.515170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:54.516219 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:54.516228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.517204 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.517208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:54.517336 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:54.517381 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 18:28:54.517508 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 18:28:54.517645 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":74,"thread_start_us":97,"threads_started":1}
I20260808 18:28:54.517812 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:54.511707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46978 (local address 127.29.78.131:37807)
0808 18:28:54.512863 (+ 1156us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:54.512881 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:54.512909 (+ 28us) server_negotiation.cc:407] Connection header received
0808 18:28:54.512987 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:54.512995 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:54.513204 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:54.513788 (+ 584us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:54.513956 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.515007 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.516343 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.517079 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.517191 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:54.517337 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:54.517543 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 18:28:54.517641 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1043,"thread_start_us":101,"threads_started":1}
I20260808 18:28:54.518236 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:54.511414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:33785 (local address 127.0.0.1:59572)
0808 18:28:54.512510 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:54.512526 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:54.512618 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:54.513606 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:54.513609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:54.513627 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:54.513732 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:54.513738 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.515092 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.515095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:54.516497 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:54.516508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.517750 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.517753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:54.517891 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:54.517921 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 18:28:54.517996 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 18:28:54.518078 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":7,"thread_start_us":71,"threads_started":1}
I20260808 18:28:54.518559 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:54.511702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:35535 (local address 127.0.0.1:54032)
0808 18:28:54.513138 (+ 1436us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:54.513152 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:54.513287 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:54.514207 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:54.514210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:54.514228 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:54.514335 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:54.514341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.515685 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.515689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:54.516754 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:54.516761 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.517847 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.517853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:54.518210 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:54.518234 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 18:28:54.518310 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 18:28:54.518385 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1348,"mutex_wait_us":24,"thread_start_us":106,"threads_started":1}
I20260808 18:28:54.518865 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:54.512199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59572 (local address 127.29.78.130:33785)
0808 18:28:54.512794 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:54.512816 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:54.512836 (+ 20us) server_negotiation.cc:407] Connection header received
0808 18:28:54.512926 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:54.512934 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:54.513157 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:54.513422 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:54.513898 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.514951 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.516631 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.517438 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.517556 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:54.517683 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:54.518621 (+ 938us) server_negotiation.cc:299] Negotiation successful
0808 18:28:54.518715 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":123,"threads_started":1}
I20260808 18:28:54.519781 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:54.512197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54032 (local address 127.29.78.129:35535)
0808 18:28:54.512619 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:54.512627 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:54.513279 (+ 652us) server_negotiation.cc:407] Connection header received
0808 18:28:54.513529 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:54.513535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:54.513802 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:54.513990 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:54.514479 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.515548 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.516865 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.517551 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.517643 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:54.517769 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:54.518814 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0808 18:28:54.518891 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":127,"threads_started":1}
I20260808 18:28:54.521126 2563 tablet_service.cc:1467] Processing CreateTablet for tablet bac968b836c9410dbed1a9b7bf3aa85a (DEFAULT_TABLE table=test-table [id=30a4d358fd62449bb111ee0f78c14b23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:54.521209 2410 tablet_service.cc:1467] Processing CreateTablet for tablet bac968b836c9410dbed1a9b7bf3aa85a (DEFAULT_TABLE table=test-table [id=30a4d358fd62449bb111ee0f78c14b23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:54.522100 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bac968b836c9410dbed1a9b7bf3aa85a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:54.522152 2563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bac968b836c9410dbed1a9b7bf3aa85a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:54.524099 2262 tablet_service.cc:1467] Processing CreateTablet for tablet bac968b836c9410dbed1a9b7bf3aa85a (DEFAULT_TABLE table=test-table [id=30a4d358fd62449bb111ee0f78c14b23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:54.525108 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bac968b836c9410dbed1a9b7bf3aa85a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:54.528790 2657 tablet_bootstrap.cc:492] T bac968b836c9410dbed1a9b7bf3aa85a P 08aec525c786476fb11c6a2a5376f113: Bootstrap starting.
I20260808 18:28:54.530763 2658 tablet_bootstrap.cc:492] T bac968b836c9410dbed1a9b7bf3aa85a P 657e59ae44ea44ffbdd3015755118d55: Bootstrap starting.
I20260808 18:28:54.531190 2657 tablet_bootstrap.cc:654] T bac968b836c9410dbed1a9b7bf3aa85a P 08aec525c786476fb11c6a2a5376f113: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:54.532114 2659 tablet_bootstrap.cc:492] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016: Bootstrap starting.
I20260808 18:28:54.533221 2658 tablet_bootstrap.cc:654] T bac968b836c9410dbed1a9b7bf3aa85a P 657e59ae44ea44ffbdd3015755118d55: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:54.533761 2657 log.cc:824] T bac968b836c9410dbed1a9b7bf3aa85a P 08aec525c786476fb11c6a2a5376f113: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:54.533931 2658 log.cc:824] T bac968b836c9410dbed1a9b7bf3aa85a P 657e59ae44ea44ffbdd3015755118d55: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:54.534554 2659 tablet_bootstrap.cc:654] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:54.535492 2659 log.cc:824] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:54.535671 2657 tablet_bootstrap.cc:492] T bac968b836c9410dbed1a9b7bf3aa85a P 08aec525c786476fb11c6a2a5376f113: No bootstrap required, opened a new log
I20260808 18:28:54.535898 2657 ts_tablet_manager.cc:1397] T bac968b836c9410dbed1a9b7bf3aa85a P 08aec525c786476fb11c6a2a5376f113: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 18:28:54.536139 2658 tablet_bootstrap.cc:492] T bac968b836c9410dbed1a9b7bf3aa85a P 657e59ae44ea44ffbdd3015755118d55: No bootstrap required, opened a new log
I20260808 18:28:54.536347 2658 ts_tablet_manager.cc:1397] T bac968b836c9410dbed1a9b7bf3aa85a P 657e59ae44ea44ffbdd3015755118d55: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:28:54.537475 2659 tablet_bootstrap.cc:492] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016: No bootstrap required, opened a new log
I20260808 18:28:54.537657 2659 ts_tablet_manager.cc:1397] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260808 18:28:54.539520 2658 raft_consensus.cc:348] T bac968b836c9410dbed1a9b7bf3aa85a P 657e59ae44ea44ffbdd3015755118d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657e59ae44ea44ffbdd3015755118d55" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33785 } } peers { permanent_uuid: "08aec525c786476fb11c6a2a5376f113" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37807 } } peers { permanent_uuid: "cf5002a4e76b4519b186c7a392ce4016" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35535 } }
I20260808 18:28:54.539852 2658 raft_consensus.cc:374] T bac968b836c9410dbed1a9b7bf3aa85a P 657e59ae44ea44ffbdd3015755118d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:54.539949 2658 raft_consensus.cc:729] T bac968b836c9410dbed1a9b7bf3aa85a P 657e59ae44ea44ffbdd3015755118d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 657e59ae44ea44ffbdd3015755118d55, State: Initialized, Role: FOLLOWER
I20260808 18:28:54.540066 2657 raft_consensus.cc:348] T bac968b836c9410dbed1a9b7bf3aa85a P 08aec525c786476fb11c6a2a5376f113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657e59ae44ea44ffbdd3015755118d55" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33785 } } peers { permanent_uuid: "08aec525c786476fb11c6a2a5376f113" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37807 } } peers { permanent_uuid: "cf5002a4e76b4519b186c7a392ce4016" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35535 } }
I20260808 18:28:54.540378 2657 raft_consensus.cc:374] T bac968b836c9410dbed1a9b7bf3aa85a P 08aec525c786476fb11c6a2a5376f113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:54.540447 2657 raft_consensus.cc:729] T bac968b836c9410dbed1a9b7bf3aa85a P 08aec525c786476fb11c6a2a5376f113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08aec525c786476fb11c6a2a5376f113, State: Initialized, Role: FOLLOWER
I20260808 18:28:54.540443 2658 consensus_queue.cc:260] T bac968b836c9410dbed1a9b7bf3aa85a P 657e59ae44ea44ffbdd3015755118d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657e59ae44ea44ffbdd3015755118d55" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33785 } } peers { permanent_uuid: "08aec525c786476fb11c6a2a5376f113" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37807 } } peers { permanent_uuid: "cf5002a4e76b4519b186c7a392ce4016" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35535 } }
I20260808 18:28:54.540905 2657 consensus_queue.cc:260] T bac968b836c9410dbed1a9b7bf3aa85a P 08aec525c786476fb11c6a2a5376f113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657e59ae44ea44ffbdd3015755118d55" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33785 } } peers { permanent_uuid: "08aec525c786476fb11c6a2a5376f113" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37807 } } peers { permanent_uuid: "cf5002a4e76b4519b186c7a392ce4016" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35535 } }
I20260808 18:28:54.541115 2488 heartbeater.cc:503] Master 127.29.78.190:41097 was elected leader, sending a full tablet report...
I20260808 18:28:54.541465 2658 ts_tablet_manager.cc:1428] T bac968b836c9410dbed1a9b7bf3aa85a P 657e59ae44ea44ffbdd3015755118d55: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 18:28:54.541911 2657 ts_tablet_manager.cc:1428] T bac968b836c9410dbed1a9b7bf3aa85a P 08aec525c786476fb11c6a2a5376f113: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 18:28:54.542162 2638 heartbeater.cc:503] Master 127.29.78.190:41097 was elected leader, sending a full tablet report...
I20260808 18:28:54.542017 2659 raft_consensus.cc:348] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657e59ae44ea44ffbdd3015755118d55" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33785 } } peers { permanent_uuid: "08aec525c786476fb11c6a2a5376f113" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37807 } } peers { permanent_uuid: "cf5002a4e76b4519b186c7a392ce4016" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35535 } }
I20260808 18:28:54.542268 2659 raft_consensus.cc:374] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:54.542328 2659 raft_consensus.cc:729] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf5002a4e76b4519b186c7a392ce4016, State: Initialized, Role: FOLLOWER
I20260808 18:28:54.542737 2659 consensus_queue.cc:260] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657e59ae44ea44ffbdd3015755118d55" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33785 } } peers { permanent_uuid: "08aec525c786476fb11c6a2a5376f113" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37807 } } peers { permanent_uuid: "cf5002a4e76b4519b186c7a392ce4016" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35535 } }
I20260808 18:28:54.543363 2336 heartbeater.cc:503] Master 127.29.78.190:41097 was elected leader, sending a full tablet report...
I20260808 18:28:54.543622 2659 ts_tablet_manager.cc:1428] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 18:28:54.546957 2666 raft_consensus.cc:482] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:28:54.547127 2666 raft_consensus.cc:504] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657e59ae44ea44ffbdd3015755118d55" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33785 } } peers { permanent_uuid: "08aec525c786476fb11c6a2a5376f113" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37807 } } peers { permanent_uuid: "cf5002a4e76b4519b186c7a392ce4016" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35535 } }
I20260808 18:28:54.548071 2666 leader_election.cc:288] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 657e59ae44ea44ffbdd3015755118d55 (127.29.78.130:33785), 08aec525c786476fb11c6a2a5376f113 (127.29.78.131:37807)
I20260808 18:28:54.552455 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:54.548718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:44993 (local address 127.29.78.130:33785)
0808 18:28:54.548827 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:54.548832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:54.548975 (+ 143us) server_negotiation.cc:407] Connection header received
0808 18:28:54.549113 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:54.549116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:54.549166 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:54.549245 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:54.549633 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.550264 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.551446 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.551899 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.551993 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:54.552031 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:54.552253 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 18:28:54.552343 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 18:28:54.552469 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:54.548263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:33785 (local address 127.29.78.129:44993)
0808 18:28:54.548885 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:54.548898 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:54.549019 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:54.549285 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:54.549288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:54.549315 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:54.549501 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:54.549508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.550366 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.550370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:54.551321 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:54.551331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.552016 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.552020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:54.552127 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:54.552152 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 18:28:54.552227 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 18:28:54.552297 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":3,"spinlock_wait_cycles":13056,"thread_start_us":172,"threads_started":1}
I20260808 18:28:54.552561 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:54.548722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:35689 (local address 127.29.78.131:37807)
0808 18:28:54.548834 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:54.548838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:54.549413 (+ 575us) server_negotiation.cc:407] Connection header received
0808 18:28:54.549513 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:54.549516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:54.549569 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:54.549642 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:28:54.549924 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.550544 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.551580 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.552007 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.552132 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:54.552176 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:54.552369 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 18:28:54.552435 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 18:28:54.552963 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:54.548298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:37807 (local address 127.29.78.129:35689)
0808 18:28:54.549253 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:54.549276 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:54.549399 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:54.549665 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:54.549668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:54.549698 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:28:54.549810 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:54.549816 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.550645 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.550647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:54.551453 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:54.551463 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.552135 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.552137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:54.552236 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:54.552259 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 18:28:54.552334 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 18:28:54.552406 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":111,"threads_started":1}
I20260808 18:28:54.553200 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac968b836c9410dbed1a9b7bf3aa85a" candidate_uuid: "cf5002a4e76b4519b186c7a392ce4016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "657e59ae44ea44ffbdd3015755118d55" is_pre_election: true
I20260808 18:28:54.553534 2435 raft_consensus.cc:2454] T bac968b836c9410dbed1a9b7bf3aa85a P 657e59ae44ea44ffbdd3015755118d55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf5002a4e76b4519b186c7a392ce4016 in term 0.
I20260808 18:28:54.553555 2588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac968b836c9410dbed1a9b7bf3aa85a" candidate_uuid: "cf5002a4e76b4519b186c7a392ce4016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08aec525c786476fb11c6a2a5376f113" is_pre_election: true
I20260808 18:28:54.553825 2588 raft_consensus.cc:2454] T bac968b836c9410dbed1a9b7bf3aa85a P 08aec525c786476fb11c6a2a5376f113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf5002a4e76b4519b186c7a392ce4016 in term 0.
I20260808 18:28:54.554034 2212 leader_election.cc:302] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 657e59ae44ea44ffbdd3015755118d55, cf5002a4e76b4519b186c7a392ce4016; no voters:
I20260808 18:28:54.554328 2666 raft_consensus.cc:2781] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:28:54.554452 2666 raft_consensus.cc:482] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:28:54.554515 2666 raft_consensus.cc:3037] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:54.555538 2666 raft_consensus.cc:504] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657e59ae44ea44ffbdd3015755118d55" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33785 } } peers { permanent_uuid: "08aec525c786476fb11c6a2a5376f113" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37807 } } peers { permanent_uuid: "cf5002a4e76b4519b186c7a392ce4016" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35535 } }
I20260808 18:28:54.555889 2666 leader_election.cc:288] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016 [CANDIDATE]: Term 1 election: Requested vote from peers 657e59ae44ea44ffbdd3015755118d55 (127.29.78.130:33785), 08aec525c786476fb11c6a2a5376f113 (127.29.78.131:37807)
I20260808 18:28:54.556200 2588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac968b836c9410dbed1a9b7bf3aa85a" candidate_uuid: "cf5002a4e76b4519b186c7a392ce4016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08aec525c786476fb11c6a2a5376f113"
I20260808 18:28:54.556226 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac968b836c9410dbed1a9b7bf3aa85a" candidate_uuid: "cf5002a4e76b4519b186c7a392ce4016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "657e59ae44ea44ffbdd3015755118d55"
I20260808 18:28:54.556305 2588 raft_consensus.cc:3037] T bac968b836c9410dbed1a9b7bf3aa85a P 08aec525c786476fb11c6a2a5376f113 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:54.556386 2435 raft_consensus.cc:3037] T bac968b836c9410dbed1a9b7bf3aa85a P 657e59ae44ea44ffbdd3015755118d55 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:54.557322 2588 raft_consensus.cc:2454] T bac968b836c9410dbed1a9b7bf3aa85a P 08aec525c786476fb11c6a2a5376f113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf5002a4e76b4519b186c7a392ce4016 in term 1.
I20260808 18:28:54.557463 2435 raft_consensus.cc:2454] T bac968b836c9410dbed1a9b7bf3aa85a P 657e59ae44ea44ffbdd3015755118d55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf5002a4e76b4519b186c7a392ce4016 in term 1.
I20260808 18:28:54.557637 2213 leader_election.cc:302] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08aec525c786476fb11c6a2a5376f113, cf5002a4e76b4519b186c7a392ce4016; no voters:
I20260808 18:28:54.557811 2666 raft_consensus.cc:2781] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:54.558068 2666 raft_consensus.cc:686] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016 [term 1 LEADER]: Becoming Leader. State: Replica: cf5002a4e76b4519b186c7a392ce4016, State: Running, Role: LEADER
I20260808 18:28:54.558387 2666 consensus_queue.cc:237] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657e59ae44ea44ffbdd3015755118d55" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33785 } } peers { permanent_uuid: "08aec525c786476fb11c6a2a5376f113" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37807 } } peers { permanent_uuid: "cf5002a4e76b4519b186c7a392ce4016" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35535 } }
I20260808 18:28:54.561497 2125 catalog_manager.cc:5495] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf5002a4e76b4519b186c7a392ce4016 (127.29.78.129). New cstate: current_term: 1 leader_uuid: "cf5002a4e76b4519b186c7a392ce4016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657e59ae44ea44ffbdd3015755118d55" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08aec525c786476fb11c6a2a5376f113" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf5002a4e76b4519b186c7a392ce4016" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35535 } health_report { overall_health: HEALTHY } } }
I20260808 18:28:54.579587 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:54.576037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54036 (local address 127.29.78.129:35535)
0808 18:28:54.576151 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:54.576155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:54.576167 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:28:54.576310 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:54.576312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:54.576361 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:54.576449 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:28:54.576686 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.577162 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.578022 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:54.578250 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:54.578338 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:54.578685 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:28:54.579062 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:28:54.579154 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:54.579221 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 18:28:54.579267 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 18:28:54.585202 2588 raft_consensus.cc:1261] T bac968b836c9410dbed1a9b7bf3aa85a P 08aec525c786476fb11c6a2a5376f113 [term 1 FOLLOWER]: Refusing update from remote peer cf5002a4e76b4519b186c7a392ce4016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:28:54.585203 2435 raft_consensus.cc:1261] T bac968b836c9410dbed1a9b7bf3aa85a P 657e59ae44ea44ffbdd3015755118d55 [term 1 FOLLOWER]: Refusing update from remote peer cf5002a4e76b4519b186c7a392ce4016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:28:54.585886 2669 consensus_queue.cc:1035] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08aec525c786476fb11c6a2a5376f113" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:28:54.586076 2666 consensus_queue.cc:1035] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "657e59ae44ea44ffbdd3015755118d55" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:28:54.593520 2673 mvcc.cc:205] Tried to move back new op lower bound from 7316331456854200320 to 7316331456754569216. Current Snapshot: MvccSnapshot[applied={T|T < 7316331456854200320 or (T in {7316331456854200320})}]
I20260808 18:28:54.594372 2675 mvcc.cc:205] Tried to move back new op lower bound from 7316331456854200320 to 7316331456754569216. Current Snapshot: MvccSnapshot[applied={T|T < 7316331456854200320}]
I20260808 18:28:54.601034 2126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:28:54.601171 2126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:28:54.603377 2126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2260a2ec58f4661990ead6020705595: Sending DeleteTablet for 3 replicas of tablet bac968b836c9410dbed1a9b7bf3aa85a
I20260808 18:28:54.604135 2262 tablet_service.cc:1514] Processing DeleteTablet for tablet bac968b836c9410dbed1a9b7bf3aa85a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:54 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:54032
I20260808 18:28:54.604138 2563 tablet_service.cc:1514] Processing DeleteTablet for tablet bac968b836c9410dbed1a9b7bf3aa85a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:54 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:46978
I20260808 18:28:54.604624 2686 tablet_replica.cc:330] T bac968b836c9410dbed1a9b7bf3aa85a P 08aec525c786476fb11c6a2a5376f113: stopping tablet replica
I20260808 18:28:54.604611 2410 tablet_service.cc:1514] Processing DeleteTablet for tablet bac968b836c9410dbed1a9b7bf3aa85a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:54 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:59572
I20260808 18:28:54.604872 2685 tablet_replica.cc:330] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016: stopping tablet replica
I20260808 18:28:54.605052 2687 tablet_replica.cc:330] T bac968b836c9410dbed1a9b7bf3aa85a P 657e59ae44ea44ffbdd3015755118d55: stopping tablet replica
I20260808 18:28:54.605340 2687 raft_consensus.cc:2229] T bac968b836c9410dbed1a9b7bf3aa85a P 657e59ae44ea44ffbdd3015755118d55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:28:54.605573 2685 raft_consensus.cc:2229] T bac968b836c9410dbed1a9b7bf3aa85a P cf5002a4e76b4519b186c7a392ce4016 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:28:54.605647 2687 raft_consensus.cc:2258] T bac968b836c9410dbed1a9b7bf3aa85a P 657e59ae44ea44ffbdd3015755118d55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:28:54.605729 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 2192
I20260808 18:28:54.607367 2687 ts_tablet_manager.cc:1905] T bac968b836c9410dbed1a9b7bf3aa85a P 657e59ae44ea44ffbdd3015755118d55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:28:54.605358 2686 raft_consensus.cc:2229] T bac968b836c9410dbed1a9b7bf3aa85a P 08aec525c786476fb11c6a2a5376f113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:28:54.609977 2687 ts_tablet_manager.cc:1918] T bac968b836c9410dbed1a9b7bf3aa85a P 657e59ae44ea44ffbdd3015755118d55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:28:54.610075 2687 log.cc:1196] T bac968b836c9410dbed1a9b7bf3aa85a P 657e59ae44ea44ffbdd3015755118d55: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/bac968b836c9410dbed1a9b7bf3aa85a
I20260808 18:28:54.610144 2686 raft_consensus.cc:2258] T bac968b836c9410dbed1a9b7bf3aa85a P 08aec525c786476fb11c6a2a5376f113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:28:54.610451 2687 ts_tablet_manager.cc:1939] T bac968b836c9410dbed1a9b7bf3aa85a P 657e59ae44ea44ffbdd3015755118d55: Deleting consensus metadata
I20260808 18:28:54.611356 2686 ts_tablet_manager.cc:1905] T bac968b836c9410dbed1a9b7bf3aa85a P 08aec525c786476fb11c6a2a5376f113: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:28:54.611642 2107 catalog_manager.cc:4841] TS 657e59ae44ea44ffbdd3015755118d55 (127.29.78.130:33785): tablet bac968b836c9410dbed1a9b7bf3aa85a (table test-table [id=30a4d358fd62449bb111ee0f78c14b23]) successfully deleted
I20260808 18:28:54.613399 2686 ts_tablet_manager.cc:1918] T bac968b836c9410dbed1a9b7bf3aa85a P 08aec525c786476fb11c6a2a5376f113: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:28:54.613477 2686 log.cc:1196] T bac968b836c9410dbed1a9b7bf3aa85a P 08aec525c786476fb11c6a2a5376f113: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/bac968b836c9410dbed1a9b7bf3aa85a
I20260808 18:28:54.613775 2686 ts_tablet_manager.cc:1939] T bac968b836c9410dbed1a9b7bf3aa85a P 08aec525c786476fb11c6a2a5376f113: Deleting consensus metadata
I20260808 18:28:54.614776 2108 catalog_manager.cc:4841] TS 08aec525c786476fb11c6a2a5376f113 (127.29.78.131:37807): tablet bac968b836c9410dbed1a9b7bf3aa85a (table test-table [id=30a4d358fd62449bb111ee0f78c14b23]) successfully deleted
I20260808 18:28:54.616287 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 2342
W20260808 18:28:54.616356 2110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:35535 (error 108)
I20260808 18:28:54.620362 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:54.620036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:35535 (local address 127.0.0.1:54044)
0808 18:28:54.620171 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:54.620254 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 18:28:54.620543 2110 catalog_manager.cc:4584] TS cf5002a4e76b4519b186c7a392ce4016 (127.29.78.129:35535): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bac968b836c9410dbed1a9b7bf3aa85a: Network error: Client connection negotiation failed: client connection to 127.29.78.129:35535: connect: Connection refused (error 111)
I20260808 18:28:54.625435 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 2493
I20260808 18:28:54.631681 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 2090
2026-08-08T18:28:54Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2284 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-08T18:28:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:28:54Z Disabled control of system clock
I20260808 18:28:54.664943 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:39347
--webserver_interface=127.29.78.190
--webserver_port=0
--builtin_ntp_servers=127.29.78.148:35049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:39347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 18:28:54.774731 2692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:54.774996 2692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:54.775034 2692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:54.778916 2692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:28:54.778967 2692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:54.778983 2692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:54.778999 2692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:28:54.779014 2692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:28:54.783598 2692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:39347
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:39347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:54.784627 2692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:54.785964 2692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:54.794516 2697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:54.794517 2699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:54.794665 2701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:54.794613 2692 server_base.cc:1034] running on GCE node
I20260808 18:28:54.795287 2692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:54.796319 2692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:54.797490 2692 hybrid_clock.cc:648] HybridClock initialized: now 1786213734797471 us; error 32 us; skew 500 ppm
I20260808 18:28:54.799378 2692 webserver.cc:460] Webserver started at http://127.29.78.190:36067/ using document root <none> and password file <none>
I20260808 18:28:54.799950 2692 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:54.800016 2692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:54.800228 2692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:54.802289 2692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "bec9bc94f0de4954b55fcf9dbee81cf9"
format_stamp: "Formatted at 2026-08-08 18:28:54 on dist-test-slave-gp6n"
server_key: "f3ee37bcc9d451299aa2b19b7fa21a5c"
server_key_iv: "ddb985ecc833bb4de9ed4cfdba411ca9"
server_key_version: "encryptionkey@0"
I20260808 18:28:54.802919 2692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "bec9bc94f0de4954b55fcf9dbee81cf9"
format_stamp: "Formatted at 2026-08-08 18:28:54 on dist-test-slave-gp6n"
server_key: "f3ee37bcc9d451299aa2b19b7fa21a5c"
server_key_iv: "ddb985ecc833bb4de9ed4cfdba411ca9"
server_key_version: "encryptionkey@0"
I20260808 18:28:54.806595 2692 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:28:54.809075 2709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:54.810562 2692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:28:54.810705 2692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "bec9bc94f0de4954b55fcf9dbee81cf9"
format_stamp: "Formatted at 2026-08-08 18:28:54 on dist-test-slave-gp6n"
server_key: "f3ee37bcc9d451299aa2b19b7fa21a5c"
server_key_iv: "ddb985ecc833bb4de9ed4cfdba411ca9"
server_key_version: "encryptionkey@0"
I20260808 18:28:54.810819 2692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:54.815783 2692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:54.816452 2692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:54.816650 2692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:54.824005 2692 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:39347
I20260808 18:28:54.824018 2768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:39347 every 8 connection(s)
I20260808 18:28:54.824993 2692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:28:54.828352 2769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:54.830505 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 2692
I20260808 18:28:54.830634 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:28:54.830958 30010 external_mini_cluster.cc:1428] Setting key d9c41d96e3fe7b03b0889bb155883076
I20260808 18:28:54.834570 2769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9: Bootstrap starting.
I20260808 18:28:54.836679 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:54.832465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.29.78.190:39347)
0808 18:28:54.833112 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:54.833127 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:54.833164 (+ 37us) server_negotiation.cc:407] Connection header received
0808 18:28:54.833842 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:54.833861 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:54.834210 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:54.834611 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:54.835457 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:54.835461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:54.835473 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:28:54.835483 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:54.835540 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:28:54.835557 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:54.835700 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:54.835773 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 18:28:54.836017 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":142,"threads_started":1}
I20260808 18:28:54.836943 2769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:54.837733 2769 log.cc:824] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:54.839810 2769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9: No bootstrap required, opened a new log
I20260808 18:28:54.843572 2769 raft_consensus.cc:348] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec9bc94f0de4954b55fcf9dbee81cf9" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39347 } }
I20260808 18:28:54.843787 2769 raft_consensus.cc:374] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:54.843822 2769 raft_consensus.cc:729] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bec9bc94f0de4954b55fcf9dbee81cf9, State: Initialized, Role: FOLLOWER
I20260808 18:28:54.844169 2769 consensus_queue.cc:260] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec9bc94f0de4954b55fcf9dbee81cf9" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39347 } }
I20260808 18:28:54.844291 2769 raft_consensus.cc:388] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:28:54.844331 2769 raft_consensus.cc:482] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:28:54.844394 2769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:54.845288 2769 raft_consensus.cc:504] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec9bc94f0de4954b55fcf9dbee81cf9" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39347 } }
I20260808 18:28:54.845599 2769 leader_election.cc:302] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bec9bc94f0de4954b55fcf9dbee81cf9; no voters:
I20260808 18:28:54.845849 2769 leader_election.cc:288] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:28:54.846033 2774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:54.846273 2774 raft_consensus.cc:686] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9 [term 1 LEADER]: Becoming Leader. State: Replica: bec9bc94f0de4954b55fcf9dbee81cf9, State: Running, Role: LEADER
I20260808 18:28:54.846623 2774 consensus_queue.cc:237] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec9bc94f0de4954b55fcf9dbee81cf9" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39347 } }
I20260808 18:28:54.847239 2769 sys_catalog.cc:564] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:28:54.848439 2776 sys_catalog.cc:455] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bec9bc94f0de4954b55fcf9dbee81cf9. Latest consensus state: current_term: 1 leader_uuid: "bec9bc94f0de4954b55fcf9dbee81cf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec9bc94f0de4954b55fcf9dbee81cf9" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39347 } } }
I20260808 18:28:54.848448 2775 sys_catalog.cc:455] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bec9bc94f0de4954b55fcf9dbee81cf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec9bc94f0de4954b55fcf9dbee81cf9" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39347 } } }
I20260808 18:28:54.848599 2776 sys_catalog.cc:458] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9 [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:54.848608 2775 sys_catalog.cc:458] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9 [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:54.849046 2783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:28:54.851909 2783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:28:54.857625 2783 catalog_manager.cc:1347] Generated new cluster ID: 742ad09acfa94e28b9bf80b33bdd4856
I20260808 18:28:54.857690 2783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:28:54.867848 2783 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:28:54.868153 2783 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:28:54.874698 2783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9: Generated new TSK 0
I20260808 18:28:54.875290 2783 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:28:54.878554 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39347
--builtin_ntp_servers=127.29.78.148:35049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 18:28:54.988196 2797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:54.988438 2797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:54.988477 2797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:54.992362 2797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:54.992405 2797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:54.992458 2797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:28:54.997048 2797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:54.997982 2797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:54.999274 2797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:55.008433 2803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:55.008469 2806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:55.008440 2804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:55.008525 2797 server_base.cc:1034] running on GCE node
I20260808 18:28:55.008996 2797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:55.009598 2797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:55.010751 2797 hybrid_clock.cc:648] HybridClock initialized: now 1786213735010732 us; error 31 us; skew 500 ppm
I20260808 18:28:55.012504 2797 webserver.cc:460] Webserver started at http://127.29.78.129:46309/ using document root <none> and password file <none>
I20260808 18:28:55.013078 2797 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:55.013144 2797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:55.013314 2797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:55.015733 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "9cd4e646a587487f959d951f1aec8f7c"
format_stamp: "Formatted at 2026-08-08 18:28:55 on dist-test-slave-gp6n"
server_key: "903d520c04b17d31a3b636a2a49a3ddb"
server_key_iv: "f27b96b604c91607103275fd68b234ae"
server_key_version: "encryptionkey@0"
I20260808 18:28:55.016393 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "9cd4e646a587487f959d951f1aec8f7c"
format_stamp: "Formatted at 2026-08-08 18:28:55 on dist-test-slave-gp6n"
server_key: "903d520c04b17d31a3b636a2a49a3ddb"
server_key_iv: "f27b96b604c91607103275fd68b234ae"
server_key_version: "encryptionkey@0"
I20260808 18:28:55.020448 2797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:28:55.023233 2813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:55.024540 2797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:28:55.024645 2797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "9cd4e646a587487f959d951f1aec8f7c"
format_stamp: "Formatted at 2026-08-08 18:28:55 on dist-test-slave-gp6n"
server_key: "903d520c04b17d31a3b636a2a49a3ddb"
server_key_iv: "f27b96b604c91607103275fd68b234ae"
server_key_version: "encryptionkey@0"
I20260808 18:28:55.024749 2797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:55.029680 2797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:55.030306 2797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:55.030519 2797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:55.031068 2797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:55.031988 2797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:55.032039 2797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:55.032080 2797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:55.032104 2797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:55.041600 2797 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:42051
I20260808 18:28:55.041623 2941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:42051 every 8 connection(s)
I20260808 18:28:55.042652 2797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:28:55.045476 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 2797
I20260808 18:28:55.045616 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:28:55.045907 30010 external_mini_cluster.cc:1428] Setting key ba1778262e9b571b899c1c888eb017f1
I20260808 18:28:55.047969 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39347
--builtin_ntp_servers=127.29.78.148:35049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 18:28:55.049739 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:55.044550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:52283 (local address 127.29.78.190:39347)
0808 18:28:55.044715 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:55.044720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:55.046015 (+ 1295us) server_negotiation.cc:407] Connection header received
0808 18:28:55.047666 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:55.047670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:55.047715 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:55.047826 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:55.049128 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:55.049132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:55.049135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:28:55.049148 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:55.049173 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:28:55.049184 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:55.049285 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:55.049521 (+ 236us) server_negotiation.cc:299] Negotiation successful
0808 18:28:55.049599 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 18:28:55.050720 2944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:55.045031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:39347 (local address 127.29.78.129:52283)
0808 18:28:55.045837 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:55.045903 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:55.047427 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:55.048012 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:55.048019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:55.048152 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:55.048856 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:28:55.048858 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:55.048904 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:28:55.048977 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:55.049283 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:55.049295 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:55.049315 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 18:28:55.049573 (+ 258us) client_negotiation.cc:240] Negotiation successful
0808 18:28:55.049832 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":149,"threads_started":1}
I20260808 18:28:55.051977 2942 heartbeater.cc:347] Connected to a master server at 127.29.78.190:39347
I20260808 18:28:55.052218 2942 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:55.052714 2942 heartbeater.cc:511] Master 127.29.78.190:39347 requested a full tablet report, sending...
I20260808 18:28:55.054497 2728 ts_manager.cc:194] Registered new tserver with Master: 9cd4e646a587487f959d951f1aec8f7c (127.29.78.129:42051)
W20260808 18:28:55.161319 2945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:55.161610 2945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:55.161684 2945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:55.165696 2945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:55.165778 2945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:55.165868 2945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:28:55.170526 2945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:55.171602 2945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:55.172978 2945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:55.182464 2953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:55.182482 2955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:55.182538 2952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:55.182560 2945 server_base.cc:1034] running on GCE node
I20260808 18:28:55.183173 2945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:55.183848 2945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:55.185052 2945 hybrid_clock.cc:648] HybridClock initialized: now 1786213735185035 us; error 26 us; skew 500 ppm
I20260808 18:28:55.187058 2945 webserver.cc:460] Webserver started at http://127.29.78.130:44267/ using document root <none> and password file <none>
I20260808 18:28:55.187654 2945 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:55.187724 2945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:55.187994 2945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:55.190202 2945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "986a66a5c8f64448923093cd646b3e51"
format_stamp: "Formatted at 2026-08-08 18:28:55 on dist-test-slave-gp6n"
server_key: "d079463f20a7459711094e9928336aba"
server_key_iv: "940b93b77f8998a2489eba255e9388b7"
server_key_version: "encryptionkey@0"
I20260808 18:28:55.191246 2945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "986a66a5c8f64448923093cd646b3e51"
format_stamp: "Formatted at 2026-08-08 18:28:55 on dist-test-slave-gp6n"
server_key: "d079463f20a7459711094e9928336aba"
server_key_iv: "940b93b77f8998a2489eba255e9388b7"
server_key_version: "encryptionkey@0"
I20260808 18:28:55.195132 2945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:28:55.197631 2961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:55.199025 2945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:28:55.199153 2945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "986a66a5c8f64448923093cd646b3e51"
format_stamp: "Formatted at 2026-08-08 18:28:55 on dist-test-slave-gp6n"
server_key: "d079463f20a7459711094e9928336aba"
server_key_iv: "940b93b77f8998a2489eba255e9388b7"
server_key_version: "encryptionkey@0"
I20260808 18:28:55.199262 2945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:55.204114 2945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:55.204720 2945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:55.204909 2945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:55.205466 2945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:55.206393 2945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:55.206549 2945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:55.206626 2945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:55.206677 2945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:55.215880 2945 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:39249
I20260808 18:28:55.215931 3082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:39249 every 8 connection(s)
I20260808 18:28:55.216843 2945 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:28:55.223145 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:55.218634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:34975 (local address 127.29.78.190:39347)
0808 18:28:55.218786 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:55.218790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:55.219838 (+ 1048us) server_negotiation.cc:407] Connection header received
0808 18:28:55.221272 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:55.221276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:55.221324 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:55.221443 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:55.222638 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:55.222642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:55.222651 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:28:55.222664 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:55.222685 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:28:55.222694 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:55.222797 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:55.222951 (+ 154us) server_negotiation.cc:299] Negotiation successful
0808 18:28:55.222999 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 18:28:55.223984 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:55.219027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:39347 (local address 127.29.78.130:34975)
0808 18:28:55.219691 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:55.219732 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:55.221053 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:55.221608 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:55.221615 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:55.221705 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:55.222316 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:28:55.222317 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:55.222365 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:28:55.222489 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:55.222780 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:55.222789 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:55.222795 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 18:28:55.222995 (+ 200us) client_negotiation.cc:240] Negotiation successful
0808 18:28:55.223237 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":111,"threads_started":1}
I20260808 18:28:55.225103 3083 heartbeater.cc:347] Connected to a master server at 127.29.78.190:39347
I20260808 18:28:55.225344 3083 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:55.225369 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 2945
I20260808 18:28:55.225481 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:28:55.225798 30010 external_mini_cluster.cc:1428] Setting key fa536c150a8d6fbd3b2364b302194090
I20260808 18:28:55.225806 3083 heartbeater.cc:511] Master 127.29.78.190:39347 requested a full tablet report, sending...
I20260808 18:28:55.226925 2728 ts_manager.cc:194] Registered new tserver with Master: 986a66a5c8f64448923093cd646b3e51 (127.29.78.130:39249)
I20260808 18:28:55.228220 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39347
--builtin_ntp_servers=127.29.78.148:35049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 18:28:55.339919 3086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:55.340204 3086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:55.340271 3086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:55.344309 3086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:55.344393 3086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:55.344480 3086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:28:55.349262 3086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:55.350356 3086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:55.351764 3086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:55.361649 3095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:55.361640 3091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:55.361640 3093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:55.362473 3086 server_base.cc:1034] running on GCE node
I20260808 18:28:55.362890 3086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:55.363607 3086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:55.364796 3086 hybrid_clock.cc:648] HybridClock initialized: now 1786213735364787 us; error 32 us; skew 500 ppm
I20260808 18:28:55.366791 3086 webserver.cc:460] Webserver started at http://127.29.78.131:33235/ using document root <none> and password file <none>
I20260808 18:28:55.367396 3086 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:55.367587 3086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:55.367842 3086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:55.369854 3086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "c9fe53ec74654bb19f4378cf13a6ebf6"
format_stamp: "Formatted at 2026-08-08 18:28:55 on dist-test-slave-gp6n"
server_key: "403e9b847d655693bb080b5fa138e369"
server_key_iv: "5f436ef783d38f5e6482869add1c5513"
server_key_version: "encryptionkey@0"
I20260808 18:28:55.370532 3086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "c9fe53ec74654bb19f4378cf13a6ebf6"
format_stamp: "Formatted at 2026-08-08 18:28:55 on dist-test-slave-gp6n"
server_key: "403e9b847d655693bb080b5fa138e369"
server_key_iv: "5f436ef783d38f5e6482869add1c5513"
server_key_version: "encryptionkey@0"
I20260808 18:28:55.374136 3086 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 18:28:55.376878 3102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:55.378098 3086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:28:55.378239 3086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "c9fe53ec74654bb19f4378cf13a6ebf6"
format_stamp: "Formatted at 2026-08-08 18:28:55 on dist-test-slave-gp6n"
server_key: "403e9b847d655693bb080b5fa138e369"
server_key_iv: "5f436ef783d38f5e6482869add1c5513"
server_key_version: "encryptionkey@0"
I20260808 18:28:55.378352 3086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:55.382833 3086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:55.383359 3086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:55.383502 3086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:55.383952 3086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:55.384769 3086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:55.384814 3086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:55.384850 3086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:55.384865 3086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:55.394706 3086 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:46465
I20260808 18:28:55.394750 3229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:46465 every 8 connection(s)
I20260808 18:28:55.395640 3086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:28:55.402205 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:55.397404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:36597 (local address 127.29.78.190:39347)
0808 18:28:55.397555 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:55.397559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:55.398879 (+ 1320us) server_negotiation.cc:407] Connection header received
0808 18:28:55.400214 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:55.400217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:55.400261 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:55.400346 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:55.401479 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:55.401482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:55.401485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:28:55.401498 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:55.401560 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:28:55.401583 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:55.401827 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:55.402008 (+ 181us) server_negotiation.cc:299] Negotiation successful
0808 18:28:55.402058 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 18:28:55.403179 3232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:55.397777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:39347 (local address 127.29.78.131:36597)
0808 18:28:55.398698 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:55.398753 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:55.400002 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:55.400542 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:55.400550 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:55.400634 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:55.401163 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:28:55.401165 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:55.401203 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:28:55.401270 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:55.401807 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:55.401817 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:55.401823 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 18:28:55.402040 (+ 217us) client_negotiation.cc:240] Negotiation successful
0808 18:28:55.402301 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":106,"threads_started":1}
I20260808 18:28:55.404537 3230 heartbeater.cc:347] Connected to a master server at 127.29.78.190:39347
I20260808 18:28:55.404785 3230 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:55.404803 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 3086
I20260808 18:28:55.404929 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:28:55.405248 30010 external_mini_cluster.cc:1428] Setting key 6a14b1ae574f7cb9912221758b12c943
I20260808 18:28:55.405269 3230 heartbeater.cc:511] Master 127.29.78.190:39347 requested a full tablet report, sending...
I20260808 18:28:55.406520 2728 ts_manager.cc:194] Registered new tserver with Master: c9fe53ec74654bb19f4378cf13a6ebf6 (127.29.78.131:46465)
I20260808 18:28:55.407729 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:28:55.411480 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:55.409815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53974 (local address 127.29.78.190:39347)
0808 18:28:55.409974 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:55.409978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:55.410112 (+ 134us) server_negotiation.cc:407] Connection header received
0808 18:28:55.410199 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:55.410201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:55.410232 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:55.410304 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:55.411055 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:55.411058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:55.411061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:28:55.411072 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:55.411092 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:28:55.411100 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:55.411178 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:55.411247 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 18:28:55.411279 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 18:28:55.414737 2727 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:28:55.418013 2727 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:28:55.430325 3243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:55.426788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:46465 (local address 127.0.0.1:51496)
0808 18:28:55.427434 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:55.427449 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:55.427527 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:55.428299 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:55.428305 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:55.428381 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:55.429023 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:28:55.429028 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:55.429052 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:28:55.429124 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:55.429933 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:55.429943 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:55.429949 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 18:28:55.430081 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 18:28:55.430159 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":122,"threads_started":1}
I20260808 18:28:55.430519 3244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:55.426604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:42051 (local address 127.0.0.1:48552)
0808 18:28:55.427147 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:55.427198 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:55.427289 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:55.429027 (+ 1738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:55.429030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:55.429047 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:55.429644 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:28:55.429646 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:55.429658 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:28:55.429689 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:55.430262 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:55.430265 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:55.430269 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 18:28:55.430323 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 18:28:55.430355 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":57,"threads_started":1}
I20260808 18:28:55.430377 3247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:55.427299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51496 (local address 127.29.78.131:46465)
0808 18:28:55.427767 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:55.427782 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:55.427804 (+ 22us) server_negotiation.cc:407] Connection header received
0808 18:28:55.427874 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:55.427882 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:55.428081 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:55.428671 (+ 590us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:55.429713 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:55.429717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:55.429726 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:28:55.429734 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:55.429792 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:28:55.429826 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:55.429941 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:55.430104 (+ 163us) server_negotiation.cc:299] Negotiation successful
0808 18:28:55.430186 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":171,"threads_started":1}
I20260808 18:28:55.431030 3246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:55.427276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:39249 (local address 127.0.0.1:48350)
0808 18:28:55.427535 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:55.427549 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:55.427623 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:55.429575 (+ 1952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:55.429578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:55.429597 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:55.430012 (+ 415us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:28:55.430017 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:55.430030 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:28:55.430062 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:55.430763 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:55.430766 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:55.430770 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 18:28:55.430847 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 18:28:55.430888 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":112,"threads_started":1}
I20260808 18:28:55.431305 3248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:55.427715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48350 (local address 127.29.78.130:39249)
0808 18:28:55.428280 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:55.428296 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:55.428364 (+ 68us) server_negotiation.cc:407] Connection header received
0808 18:28:55.428459 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:55.428465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:55.428669 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:55.428817 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:55.430202 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:55.430208 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:55.430220 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:28:55.430232 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:55.430273 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:28:55.430647 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:55.430798 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:55.430959 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 18:28:55.431035 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":179,"threads_started":1}
I20260808 18:28:55.431535 3245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:55.426781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48552 (local address 127.29.78.129:42051)
0808 18:28:55.428021 (+ 1240us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:55.428031 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:55.428054 (+ 23us) server_negotiation.cc:407] Connection header received
0808 18:28:55.428247 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:55.428256 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:55.428518 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:55.428670 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:55.430030 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:55.430036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:55.430053 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:28:55.430069 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:55.430114 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:28:55.430149 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:55.431287 (+ 1138us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:55.431331 (+ 44us) server_negotiation.cc:299] Negotiation successful
0808 18:28:55.431393 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1142,"thread_start_us":176,"threads_started":1}
I20260808 18:28:55.433589 3155 tablet_service.cc:1467] Processing CreateTablet for tablet 40001ee8c9844e57a9b93bbda0a73305 (DEFAULT_TABLE table=test-table [id=0090d0c9c1354a9b80a0d6d530f9b5be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:55.434168 2868 tablet_service.cc:1467] Processing CreateTablet for tablet 40001ee8c9844e57a9b93bbda0a73305 (DEFAULT_TABLE table=test-table [id=0090d0c9c1354a9b80a0d6d530f9b5be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:55.434167 3012 tablet_service.cc:1467] Processing CreateTablet for tablet 40001ee8c9844e57a9b93bbda0a73305 (DEFAULT_TABLE table=test-table [id=0090d0c9c1354a9b80a0d6d530f9b5be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:28:55.434774 3155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40001ee8c9844e57a9b93bbda0a73305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:55.435287 3012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40001ee8c9844e57a9b93bbda0a73305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:55.435287 2868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40001ee8c9844e57a9b93bbda0a73305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:55.440869 3249 tablet_bootstrap.cc:492] T 40001ee8c9844e57a9b93bbda0a73305 P c9fe53ec74654bb19f4378cf13a6ebf6: Bootstrap starting.
I20260808 18:28:55.443236 3251 tablet_bootstrap.cc:492] T 40001ee8c9844e57a9b93bbda0a73305 P 9cd4e646a587487f959d951f1aec8f7c: Bootstrap starting.
I20260808 18:28:55.443290 3249 tablet_bootstrap.cc:654] T 40001ee8c9844e57a9b93bbda0a73305 P c9fe53ec74654bb19f4378cf13a6ebf6: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:55.444087 3249 log.cc:824] T 40001ee8c9844e57a9b93bbda0a73305 P c9fe53ec74654bb19f4378cf13a6ebf6: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:55.444792 3250 tablet_bootstrap.cc:492] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51: Bootstrap starting.
I20260808 18:28:55.445164 3251 tablet_bootstrap.cc:654] T 40001ee8c9844e57a9b93bbda0a73305 P 9cd4e646a587487f959d951f1aec8f7c: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:55.445875 3251 log.cc:824] T 40001ee8c9844e57a9b93bbda0a73305 P 9cd4e646a587487f959d951f1aec8f7c: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:55.446820 3250 tablet_bootstrap.cc:654] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:55.447715 3250 log.cc:824] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:55.447849 3249 tablet_bootstrap.cc:492] T 40001ee8c9844e57a9b93bbda0a73305 P c9fe53ec74654bb19f4378cf13a6ebf6: No bootstrap required, opened a new log
I20260808 18:28:55.447849 3251 tablet_bootstrap.cc:492] T 40001ee8c9844e57a9b93bbda0a73305 P 9cd4e646a587487f959d951f1aec8f7c: No bootstrap required, opened a new log
I20260808 18:28:55.448028 3251 ts_tablet_manager.cc:1397] T 40001ee8c9844e57a9b93bbda0a73305 P 9cd4e646a587487f959d951f1aec8f7c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 18:28:55.448028 3249 ts_tablet_manager.cc:1397] T 40001ee8c9844e57a9b93bbda0a73305 P c9fe53ec74654bb19f4378cf13a6ebf6: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 18:28:55.449824 3250 tablet_bootstrap.cc:492] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51: No bootstrap required, opened a new log
I20260808 18:28:55.450023 3250 ts_tablet_manager.cc:1397] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 18:28:55.452775 3249 raft_consensus.cc:348] T 40001ee8c9844e57a9b93bbda0a73305 P c9fe53ec74654bb19f4378cf13a6ebf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd4e646a587487f959d951f1aec8f7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42051 } } peers { permanent_uuid: "c9fe53ec74654bb19f4378cf13a6ebf6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46465 } } peers { permanent_uuid: "986a66a5c8f64448923093cd646b3e51" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39249 } }
I20260808 18:28:55.452775 3251 raft_consensus.cc:348] T 40001ee8c9844e57a9b93bbda0a73305 P 9cd4e646a587487f959d951f1aec8f7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd4e646a587487f959d951f1aec8f7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42051 } } peers { permanent_uuid: "c9fe53ec74654bb19f4378cf13a6ebf6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46465 } } peers { permanent_uuid: "986a66a5c8f64448923093cd646b3e51" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39249 } }
I20260808 18:28:55.453076 3249 raft_consensus.cc:374] T 40001ee8c9844e57a9b93bbda0a73305 P c9fe53ec74654bb19f4378cf13a6ebf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:55.453076 3251 raft_consensus.cc:374] T 40001ee8c9844e57a9b93bbda0a73305 P 9cd4e646a587487f959d951f1aec8f7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:55.453141 3251 raft_consensus.cc:729] T 40001ee8c9844e57a9b93bbda0a73305 P 9cd4e646a587487f959d951f1aec8f7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd4e646a587487f959d951f1aec8f7c, State: Initialized, Role: FOLLOWER
I20260808 18:28:55.453151 3249 raft_consensus.cc:729] T 40001ee8c9844e57a9b93bbda0a73305 P c9fe53ec74654bb19f4378cf13a6ebf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9fe53ec74654bb19f4378cf13a6ebf6, State: Initialized, Role: FOLLOWER
I20260808 18:28:55.453236 3250 raft_consensus.cc:348] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd4e646a587487f959d951f1aec8f7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42051 } } peers { permanent_uuid: "c9fe53ec74654bb19f4378cf13a6ebf6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46465 } } peers { permanent_uuid: "986a66a5c8f64448923093cd646b3e51" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39249 } }
I20260808 18:28:55.453500 3250 raft_consensus.cc:374] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:55.453562 3250 raft_consensus.cc:729] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 986a66a5c8f64448923093cd646b3e51, State: Initialized, Role: FOLLOWER
I20260808 18:28:55.453642 3249 consensus_queue.cc:260] T 40001ee8c9844e57a9b93bbda0a73305 P c9fe53ec74654bb19f4378cf13a6ebf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd4e646a587487f959d951f1aec8f7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42051 } } peers { permanent_uuid: "c9fe53ec74654bb19f4378cf13a6ebf6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46465 } } peers { permanent_uuid: "986a66a5c8f64448923093cd646b3e51" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39249 } }
I20260808 18:28:55.453641 3251 consensus_queue.cc:260] T 40001ee8c9844e57a9b93bbda0a73305 P 9cd4e646a587487f959d951f1aec8f7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd4e646a587487f959d951f1aec8f7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42051 } } peers { permanent_uuid: "c9fe53ec74654bb19f4378cf13a6ebf6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46465 } } peers { permanent_uuid: "986a66a5c8f64448923093cd646b3e51" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39249 } }
I20260808 18:28:55.453987 3250 consensus_queue.cc:260] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd4e646a587487f959d951f1aec8f7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42051 } } peers { permanent_uuid: "c9fe53ec74654bb19f4378cf13a6ebf6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46465 } } peers { permanent_uuid: "986a66a5c8f64448923093cd646b3e51" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39249 } }
I20260808 18:28:55.454535 3230 heartbeater.cc:503] Master 127.29.78.190:39347 was elected leader, sending a full tablet report...
I20260808 18:28:55.454617 2942 heartbeater.cc:503] Master 127.29.78.190:39347 was elected leader, sending a full tablet report...
I20260808 18:28:55.454793 3249 ts_tablet_manager.cc:1428] T 40001ee8c9844e57a9b93bbda0a73305 P c9fe53ec74654bb19f4378cf13a6ebf6: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260808 18:28:55.454952 3250 ts_tablet_manager.cc:1428] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 18:28:55.455231 3251 ts_tablet_manager.cc:1428] T 40001ee8c9844e57a9b93bbda0a73305 P 9cd4e646a587487f959d951f1aec8f7c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260808 18:28:55.455232 3083 heartbeater.cc:503] Master 127.29.78.190:39347 was elected leader, sending a full tablet report...
W20260808 18:28:55.467808 3084 tablet.cc:2369] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 18:28:55.543959 2943 tablet.cc:2369] T 40001ee8c9844e57a9b93bbda0a73305 P 9cd4e646a587487f959d951f1aec8f7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:28:55.639156 3259 raft_consensus.cc:482] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:28:55.639398 3259 raft_consensus.cc:504] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd4e646a587487f959d951f1aec8f7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42051 } } peers { permanent_uuid: "c9fe53ec74654bb19f4378cf13a6ebf6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46465 } } peers { permanent_uuid: "986a66a5c8f64448923093cd646b3e51" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39249 } }
I20260808 18:28:55.640671 3259 leader_election.cc:288] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cd4e646a587487f959d951f1aec8f7c (127.29.78.129:42051), c9fe53ec74654bb19f4378cf13a6ebf6 (127.29.78.131:46465)
I20260808 18:28:55.642730 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:55.640908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:42051 (local address 127.29.78.130:34395)
0808 18:28:55.641025 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:55.641043 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:55.641139 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:55.641494 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:55.641498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:55.641514 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:55.642021 (+ 507us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:28:55.642026 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:55.642042 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:28:55.642079 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:55.642424 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:55.642428 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:55.642432 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 18:28:55.642523 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 18:28:55.642564 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260808 18:28:55.642732 3245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:55.641014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:34395 (local address 127.29.78.129:42051)
0808 18:28:55.641176 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:55.641184 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:55.641206 (+ 22us) server_negotiation.cc:407] Connection header received
0808 18:28:55.641263 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:55.641266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:55.641309 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:55.641477 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:55.642221 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:55.642224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:55.642226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:28:55.642238 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:55.642259 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:28:55.642267 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:55.642434 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:55.642538 (+ 104us) server_negotiation.cc:299] Negotiation successful
0808 18:28:55.642587 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 18:28:55.642978 3247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:55.641233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:46957 (local address 127.29.78.131:46465)
0808 18:28:55.641356 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:55.641360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:55.641507 (+ 147us) server_negotiation.cc:407] Connection header received
0808 18:28:55.641635 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:55.641638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:55.641678 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:55.641764 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:55.642537 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:55.642541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:55.642545 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:28:55.642557 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:55.642580 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:28:55.642590 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:55.642691 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:55.642791 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 18:28:55.642836 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 18:28:55.642992 3260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:55.641027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:46465 (local address 127.29.78.130:46957)
0808 18:28:55.641414 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:55.641425 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:55.641529 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:55.641825 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:55.641827 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:55.641841 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:55.642326 (+ 485us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:28:55.642328 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:55.642339 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:28:55.642372 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:55.642689 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:55.642692 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:55.642695 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 18:28:55.642782 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 18:28:55.642817 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":159,"threads_started":1}
I20260808 18:28:55.643424 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40001ee8c9844e57a9b93bbda0a73305" candidate_uuid: "986a66a5c8f64448923093cd646b3e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cd4e646a587487f959d951f1aec8f7c" is_pre_election: true
I20260808 18:28:55.643750 2892 raft_consensus.cc:2454] T 40001ee8c9844e57a9b93bbda0a73305 P 9cd4e646a587487f959d951f1aec8f7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 986a66a5c8f64448923093cd646b3e51 in term 0.
I20260808 18:28:55.643713 3176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40001ee8c9844e57a9b93bbda0a73305" candidate_uuid: "986a66a5c8f64448923093cd646b3e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9fe53ec74654bb19f4378cf13a6ebf6" is_pre_election: true
I20260808 18:28:55.643986 3176 raft_consensus.cc:2454] T 40001ee8c9844e57a9b93bbda0a73305 P c9fe53ec74654bb19f4378cf13a6ebf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 986a66a5c8f64448923093cd646b3e51 in term 0.
I20260808 18:28:55.644248 2965 leader_election.cc:302] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 986a66a5c8f64448923093cd646b3e51, 9cd4e646a587487f959d951f1aec8f7c; no voters:
I20260808 18:28:55.644546 3259 raft_consensus.cc:2781] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:28:55.644630 3259 raft_consensus.cc:482] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:28:55.644673 3259 raft_consensus.cc:3037] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:55.645699 3259 raft_consensus.cc:504] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd4e646a587487f959d951f1aec8f7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42051 } } peers { permanent_uuid: "c9fe53ec74654bb19f4378cf13a6ebf6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46465 } } peers { permanent_uuid: "986a66a5c8f64448923093cd646b3e51" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39249 } }
I20260808 18:28:55.646023 3259 leader_election.cc:288] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51 [CANDIDATE]: Term 1 election: Requested vote from peers 9cd4e646a587487f959d951f1aec8f7c (127.29.78.129:42051), c9fe53ec74654bb19f4378cf13a6ebf6 (127.29.78.131:46465)
I20260808 18:28:55.646363 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40001ee8c9844e57a9b93bbda0a73305" candidate_uuid: "986a66a5c8f64448923093cd646b3e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cd4e646a587487f959d951f1aec8f7c"
I20260808 18:28:55.646482 3176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40001ee8c9844e57a9b93bbda0a73305" candidate_uuid: "986a66a5c8f64448923093cd646b3e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9fe53ec74654bb19f4378cf13a6ebf6"
W20260808 18:28:55.646572 3231 tablet.cc:2369] T 40001ee8c9844e57a9b93bbda0a73305 P c9fe53ec74654bb19f4378cf13a6ebf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:28:55.646566 2892 raft_consensus.cc:3037] T 40001ee8c9844e57a9b93bbda0a73305 P 9cd4e646a587487f959d951f1aec8f7c [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:55.646611 3176 raft_consensus.cc:3037] T 40001ee8c9844e57a9b93bbda0a73305 P c9fe53ec74654bb19f4378cf13a6ebf6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:55.647614 2892 raft_consensus.cc:2454] T 40001ee8c9844e57a9b93bbda0a73305 P 9cd4e646a587487f959d951f1aec8f7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 986a66a5c8f64448923093cd646b3e51 in term 1.
I20260808 18:28:55.647743 3176 raft_consensus.cc:2454] T 40001ee8c9844e57a9b93bbda0a73305 P c9fe53ec74654bb19f4378cf13a6ebf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 986a66a5c8f64448923093cd646b3e51 in term 1.
I20260808 18:28:55.647954 2965 leader_election.cc:302] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 986a66a5c8f64448923093cd646b3e51, 9cd4e646a587487f959d951f1aec8f7c; no voters:
I20260808 18:28:55.648180 3259 raft_consensus.cc:2781] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:55.648419 3259 raft_consensus.cc:686] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51 [term 1 LEADER]: Becoming Leader. State: Replica: 986a66a5c8f64448923093cd646b3e51, State: Running, Role: LEADER
I20260808 18:28:55.648736 3259 consensus_queue.cc:237] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd4e646a587487f959d951f1aec8f7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42051 } } peers { permanent_uuid: "c9fe53ec74654bb19f4378cf13a6ebf6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46465 } } peers { permanent_uuid: "986a66a5c8f64448923093cd646b3e51" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39249 } }
I20260808 18:28:55.651942 2727 catalog_manager.cc:5495] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 986a66a5c8f64448923093cd646b3e51 (127.29.78.130). New cstate: current_term: 1 leader_uuid: "986a66a5c8f64448923093cd646b3e51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd4e646a587487f959d951f1aec8f7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9fe53ec74654bb19f4378cf13a6ebf6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "986a66a5c8f64448923093cd646b3e51" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39249 } health_report { overall_health: HEALTHY } } }
I20260808 18:28:55.665266 3248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:55.663546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48364 (local address 127.29.78.130:39249)
0808 18:28:55.663696 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:55.663701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:55.663717 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:28:55.663914 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:55.663917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:55.663979 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:55.664077 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:55.664819 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:55.664823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:55.664826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:28:55.664857 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:55.664876 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:28:55.664885 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:55.664993 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:55.665071 (+ 78us) server_negotiation.cc:299] Negotiation successful
0808 18:28:55.665117 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 18:28:55.671060 2892 raft_consensus.cc:1261] T 40001ee8c9844e57a9b93bbda0a73305 P 9cd4e646a587487f959d951f1aec8f7c [term 1 FOLLOWER]: Refusing update from remote peer 986a66a5c8f64448923093cd646b3e51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:28:55.671083 3176 raft_consensus.cc:1261] T 40001ee8c9844e57a9b93bbda0a73305 P c9fe53ec74654bb19f4378cf13a6ebf6 [term 1 FOLLOWER]: Refusing update from remote peer 986a66a5c8f64448923093cd646b3e51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:28:55.671752 3259 consensus_queue.cc:1035] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9fe53ec74654bb19f4378cf13a6ebf6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:28:55.671980 3261 consensus_queue.cc:1035] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cd4e646a587487f959d951f1aec8f7c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:28:55.689427 2728 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:28:55.689625 2728 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:28:55.692101 2728 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bec9bc94f0de4954b55fcf9dbee81cf9: Sending DeleteTablet for 3 replicas of tablet 40001ee8c9844e57a9b93bbda0a73305
I20260808 18:28:55.692843 2868 tablet_service.cc:1514] Processing DeleteTablet for tablet 40001ee8c9844e57a9b93bbda0a73305 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:55 UTC) from {username='slave'} at 127.0.0.1:48552
I20260808 18:28:55.692873 3155 tablet_service.cc:1514] Processing DeleteTablet for tablet 40001ee8c9844e57a9b93bbda0a73305 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:55 UTC) from {username='slave'} at 127.0.0.1:51496
I20260808 18:28:55.693006 3012 tablet_service.cc:1514] Processing DeleteTablet for tablet 40001ee8c9844e57a9b93bbda0a73305 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:28:55 UTC) from {username='slave'} at 127.0.0.1:48350
I20260808 18:28:55.693347 3276 tablet_replica.cc:330] T 40001ee8c9844e57a9b93bbda0a73305 P c9fe53ec74654bb19f4378cf13a6ebf6: stopping tablet replica
I20260808 18:28:55.693375 3277 tablet_replica.cc:330] T 40001ee8c9844e57a9b93bbda0a73305 P 9cd4e646a587487f959d951f1aec8f7c: stopping tablet replica
I20260808 18:28:55.693675 3277 raft_consensus.cc:2229] T 40001ee8c9844e57a9b93bbda0a73305 P 9cd4e646a587487f959d951f1aec8f7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:28:55.693766 3278 tablet_replica.cc:330] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51: stopping tablet replica
I20260808 18:28:55.693996 3277 raft_consensus.cc:2258] T 40001ee8c9844e57a9b93bbda0a73305 P 9cd4e646a587487f959d951f1aec8f7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:28:55.694061 3278 raft_consensus.cc:2229] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:28:55.694332 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 2797
I20260808 18:28:55.694979 3278 raft_consensus.cc:2258] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:28:55.696951 3278 ts_tablet_manager.cc:1905] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:28:55.700006 3278 ts_tablet_manager.cc:1918] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:28:55.700110 3278 log.cc:1196] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/40001ee8c9844e57a9b93bbda0a73305
I20260808 18:28:55.700436 3276 raft_consensus.cc:2229] T 40001ee8c9844e57a9b93bbda0a73305 P c9fe53ec74654bb19f4378cf13a6ebf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:28:55.700470 3278 ts_tablet_manager.cc:1939] T 40001ee8c9844e57a9b93bbda0a73305 P 986a66a5c8f64448923093cd646b3e51: Deleting consensus metadata
I20260808 18:28:55.700847 3276 raft_consensus.cc:2258] T 40001ee8c9844e57a9b93bbda0a73305 P c9fe53ec74654bb19f4378cf13a6ebf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:28:55.701603 2713 catalog_manager.cc:4841] TS 986a66a5c8f64448923093cd646b3e51 (127.29.78.130:39249): tablet 40001ee8c9844e57a9b93bbda0a73305 (table test-table [id=0090d0c9c1354a9b80a0d6d530f9b5be]) successfully deleted
I20260808 18:28:55.702582 3276 ts_tablet_manager.cc:1905] T 40001ee8c9844e57a9b93bbda0a73305 P c9fe53ec74654bb19f4378cf13a6ebf6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 18:28:55.703579 2714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:42051 (error 108)
I20260808 18:28:55.704627 3246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:55.704312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:42051 (local address 127.0.0.1:48562)
0808 18:28:55.704459 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:55.704537 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 18:28:55.704705 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 2945
I20260808 18:28:55.705880 3276 ts_tablet_manager.cc:1918] T 40001ee8c9844e57a9b93bbda0a73305 P c9fe53ec74654bb19f4378cf13a6ebf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:28:55.705981 3276 log.cc:1196] T 40001ee8c9844e57a9b93bbda0a73305 P c9fe53ec74654bb19f4378cf13a6ebf6: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/40001ee8c9844e57a9b93bbda0a73305
I20260808 18:28:55.706295 3276 ts_tablet_manager.cc:1939] T 40001ee8c9844e57a9b93bbda0a73305 P c9fe53ec74654bb19f4378cf13a6ebf6: Deleting consensus metadata
W20260808 18:28:55.706519 2714 catalog_manager.cc:4584] TS 9cd4e646a587487f959d951f1aec8f7c (127.29.78.129:42051): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 40001ee8c9844e57a9b93bbda0a73305: Network error: Client connection negotiation failed: client connection to 127.29.78.129:42051: connect: Connection refused (error 111)
I20260808 18:28:55.710110 2710 catalog_manager.cc:4841] TS c9fe53ec74654bb19f4378cf13a6ebf6 (127.29.78.131:46465): tablet 40001ee8c9844e57a9b93bbda0a73305 (table test-table [id=0090d0c9c1354a9b80a0d6d530f9b5be]) successfully deleted
I20260808 18:28:55.713227 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 3086
I20260808 18:28:55.720381 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 2692
2026-08-08T18:28:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1088 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260808 18:28:55.744692 30010 mini_oidc.cc:152] Starting JWKS server
I20260808 18:28:55.746591 30010 webserver.cc:460] Webserver started at https://127.0.0.1:35655/ using document root <none> and password file <none>
I20260808 18:28:55.746781 30010 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 18:28:55.747077 30010 webserver.cc:460] Webserver started at http://127.0.0.1:43845/ using document root <none> and password file <none>
2026-08-08T18:28:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:28:55Z Disabled control of system clock
I20260808 18:28:55.757205 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:37469
--webserver_interface=127.29.78.190
--webserver_port=0
--builtin_ntp_servers=127.29.78.148:34377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:37469
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35655/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786213708769829-30010-0/testchainca.cert with env {}
W20260808 18:28:55.868613 3286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:55.868852 3286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:55.868889 3286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:55.872893 3286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:28:55.872949 3286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:55.872965 3286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:55.872982 3286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:28:55.872997 3286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 18:28:55.873013 3286 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 18:28:55.873026 3286 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35655/jwks.json
I20260808 18:28:55.878196 3286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:34377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:37469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:37469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35655/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786213708769829-30010-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:55.879211 3286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:55.880147 3286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:55.888778 3292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:55.888794 3294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:55.888789 3291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:55.889083 3286 server_base.cc:1034] running on GCE node
I20260808 18:28:55.889635 3286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:55.890784 3286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:55.891979 3286 hybrid_clock.cc:648] HybridClock initialized: now 1786213735891965 us; error 27 us; skew 500 ppm
I20260808 18:28:55.893851 3286 webserver.cc:460] Webserver started at http://127.29.78.190:36537/ using document root <none> and password file <none>
I20260808 18:28:55.894493 3286 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:55.894585 3286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:55.894815 3286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:55.896797 3286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "f1272b567dbf4e9690bbccc607e5c1ae"
format_stamp: "Formatted at 2026-08-08 18:28:55 on dist-test-slave-gp6n"
server_key: "8ca29924d8b4ad11ae8bae76734317bb"
server_key_iv: "cf0a96aa8bc54daf92ef4e9af26c0edc"
server_key_version: "encryptionkey@0"
I20260808 18:28:55.897384 3286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "f1272b567dbf4e9690bbccc607e5c1ae"
format_stamp: "Formatted at 2026-08-08 18:28:55 on dist-test-slave-gp6n"
server_key: "8ca29924d8b4ad11ae8bae76734317bb"
server_key_iv: "cf0a96aa8bc54daf92ef4e9af26c0edc"
server_key_version: "encryptionkey@0"
I20260808 18:28:55.901018 3286 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 18:28:55.903472 3300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:55.904817 3286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:28:55.904958 3286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "f1272b567dbf4e9690bbccc607e5c1ae"
format_stamp: "Formatted at 2026-08-08 18:28:55 on dist-test-slave-gp6n"
server_key: "8ca29924d8b4ad11ae8bae76734317bb"
server_key_iv: "cf0a96aa8bc54daf92ef4e9af26c0edc"
server_key_version: "encryptionkey@0"
I20260808 18:28:55.905066 3286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:55.926259 3286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:55.927047 3286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:55.927265 3286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:55.934140 3286 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:37469
I20260808 18:28:55.934190 3361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:37469 every 8 connection(s)
I20260808 18:28:55.935214 3286 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:28:55.938239 3362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:55.943692 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 3286
I20260808 18:28:55.943806 3362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae: Bootstrap starting.
I20260808 18:28:55.943843 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:28:55.944173 30010 external_mini_cluster.cc:1428] Setting key a688b30ef29e873b84a1845c59693d91
I20260808 18:28:55.946485 3362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:55.947257 3362 log.cc:824] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:55.949074 3362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae: No bootstrap required, opened a new log
I20260808 18:28:55.952548 3362 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1272b567dbf4e9690bbccc607e5c1ae" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37469 } }
I20260808 18:28:55.952813 3362 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:55.952890 3362 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1272b567dbf4e9690bbccc607e5c1ae, State: Initialized, Role: FOLLOWER
I20260808 18:28:55.953362 3362 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1272b567dbf4e9690bbccc607e5c1ae" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37469 } }
I20260808 18:28:55.953490 3362 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:28:55.953573 3362 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:28:55.953577 3366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:55.945448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.29.78.190:37469)
0808 18:28:55.946106 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:55.946120 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:55.946155 (+ 35us) server_negotiation.cc:407] Connection header received
0808 18:28:55.946950 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:55.946969 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:55.947451 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:55.947774 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:55.948194 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:55.949148 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:55.950100 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:55.950649 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:55.950825 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:55.951708 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:55.951713 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:55.951725 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:28:55.952102 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:55.952158 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:28:55.952178 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:55.952286 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:55.952405 (+ 119us) server_negotiation.cc:299] Negotiation successful
0808 18:28:55.952670 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":148,"threads_started":1}
I20260808 18:28:55.953699 3362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:55.954629 3362 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1272b567dbf4e9690bbccc607e5c1ae" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37469 } }
I20260808 18:28:55.954975 3362 leader_election.cc:302] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1272b567dbf4e9690bbccc607e5c1ae; no voters:
I20260808 18:28:55.955286 3362 leader_election.cc:288] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:28:55.955399 3368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:55.955626 3368 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae [term 1 LEADER]: Becoming Leader. State: Replica: f1272b567dbf4e9690bbccc607e5c1ae, State: Running, Role: LEADER
I20260808 18:28:55.955925 3368 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1272b567dbf4e9690bbccc607e5c1ae" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37469 } }
I20260808 18:28:55.956526 3362 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:28:55.956720 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:28:55.957532 3370 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1272b567dbf4e9690bbccc607e5c1ae. Latest consensus state: current_term: 1 leader_uuid: "f1272b567dbf4e9690bbccc607e5c1ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1272b567dbf4e9690bbccc607e5c1ae" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37469 } } }
I20260808 18:28:55.957672 3370 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:55.957969 3369 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1272b567dbf4e9690bbccc607e5c1ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1272b567dbf4e9690bbccc607e5c1ae" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37469 } } }
I20260808 18:28:55.958062 3369 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae [sys.catalog]: This master's current role is: LEADER
W20260808 18:28:55.961426 3385 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:28:55.961522 3385 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:28:55.961685 3379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:28:55.962718 3379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:28:55.967996 3379 catalog_manager.cc:1347] Generated new cluster ID: 76839c54e1d548309ca0c7342e49c9d1
I20260808 18:28:55.968061 3379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:28:55.988917 3379 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:28:55.990127 3379 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:28:56.002172 3379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1272b567dbf4e9690bbccc607e5c1ae: Generated new TSK 0
I20260808 18:28:56.002988 3379 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 18:28:56.035202 3396 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260808 18:28:56.035439 3396 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 18:28:56.036069 3366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:56.017022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.29.78.190:37469)
0808 18:28:56.017190 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:56.017195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:56.017473 (+ 278us) server_negotiation.cc:407] Connection header received
0808 18:28:56.017676 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:56.017678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:56.017728 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:56.017821 (+ 93us) server_negotiation.cc:226] Negotiated authn=JWT
0808 18:28:56.018123 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:56.018883 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:56.020123 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:56.020385 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:56.020474 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:56.020829 (+ 355us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 18:28:56.035640 (+ 14811us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 18:28:56.035949 (+ 309us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40262: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260808 18:28:56.036175 3366 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40262: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260808 18:28:56.037710 3395 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:28:56.016961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:37469 (local address 127.0.0.1:40262)
0808 18:28:56.017352 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:56.017368 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:56.017510 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:56.017858 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:56.017861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:56.017869 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0808 18:28:56.017987 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:56.017993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:56.018995 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:56.018999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:56.019997 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:56.020007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:56.020501 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:56.020505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:56.020593 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:56.020642 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 18:28:56.035913 (+ 15271us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 18:28:56.036177 (+ 264us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.78.190:37469: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":160,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.29.78.190:37469: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260808 18:28:56.038511 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 3286
2026-08-08T18:28:56Z chronyd exiting
I20260808 18:28:56.078761 30010 test_util.cc:175] -----------------------------------------------
I20260808 18:28:56.078852 30010 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786213708769829-30010-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (335 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260808 18:28:56.079901 30010 mini_oidc.cc:152] Starting JWKS server
I20260808 18:28:56.080727 30010 webserver.cc:460] Webserver started at https://127.0.0.1:40769/ using document root <none> and password file <none>
I20260808 18:28:56.080804 30010 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 18:28:56.081017 30010 webserver.cc:460] Webserver started at http://127.0.0.1:39423/ using document root <none> and password file <none>
2026-08-08T18:28:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:28:56Z Disabled control of system clock
I20260808 18:28:56.090904 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:36491
--webserver_interface=127.29.78.190
--webserver_port=0
--builtin_ntp_servers=127.29.78.148:39731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:36491
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40769/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786213708769829-30010-0/testchainca.cert with env {}
W20260808 18:28:56.201598 3403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:56.201856 3403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:56.201895 3403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:56.206486 3403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:28:56.206544 3403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:56.206559 3403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:56.206578 3403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:28:56.206593 3403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 18:28:56.206609 3403 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 18:28:56.206621 3403 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40769/jwks.json
I20260808 18:28:56.211843 3403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:36491
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:36491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40769/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786213708769829-30010-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:56.212787 3403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:56.213732 3403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:56.224433 3413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:56.224539 3408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:56.224563 3409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:56.224970 3403 server_base.cc:1034] running on GCE node
I20260808 18:28:56.225526 3403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:56.226745 3403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:56.227946 3403 hybrid_clock.cc:648] HybridClock initialized: now 1786213736227929 us; error 31 us; skew 500 ppm
I20260808 18:28:56.229830 3403 webserver.cc:460] Webserver started at http://127.29.78.190:46495/ using document root <none> and password file <none>
I20260808 18:28:56.230479 3403 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:56.230577 3403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:56.230813 3403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:56.232877 3403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "ceaaf42868514b20a3aef4fe58107cc0"
format_stamp: "Formatted at 2026-08-08 18:28:56 on dist-test-slave-gp6n"
server_key: "41d1d062794c22ecff1f57b7c7018546"
server_key_iv: "71b92a4fc4ed2c3cad2b152079828143"
server_key_version: "encryptionkey@0"
I20260808 18:28:56.233481 3403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "ceaaf42868514b20a3aef4fe58107cc0"
format_stamp: "Formatted at 2026-08-08 18:28:56 on dist-test-slave-gp6n"
server_key: "41d1d062794c22ecff1f57b7c7018546"
server_key_iv: "71b92a4fc4ed2c3cad2b152079828143"
server_key_version: "encryptionkey@0"
I20260808 18:28:56.237236 3403 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260808 18:28:56.239775 3420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:56.241148 3403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:28:56.241287 3403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "ceaaf42868514b20a3aef4fe58107cc0"
format_stamp: "Formatted at 2026-08-08 18:28:56 on dist-test-slave-gp6n"
server_key: "41d1d062794c22ecff1f57b7c7018546"
server_key_iv: "71b92a4fc4ed2c3cad2b152079828143"
server_key_version: "encryptionkey@0"
I20260808 18:28:56.241400 3403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:56.250669 3403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:56.251281 3403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:56.251467 3403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:56.258617 3403 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:36491
I20260808 18:28:56.258675 3477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:36491 every 8 connection(s)
I20260808 18:28:56.259552 3403 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:28:56.262910 3478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:56.266834 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 3403
I20260808 18:28:56.266986 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:28:56.267323 30010 external_mini_cluster.cc:1428] Setting key 6bfbfa48536608c6d5357d9ded2baf6c
I20260808 18:28:56.269124 3478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0: Bootstrap starting.
I20260808 18:28:56.271384 3478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:56.272132 3478 log.cc:824] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:56.273939 3478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0: No bootstrap required, opened a new log
I20260808 18:28:56.276497 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:56.268707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34562 (local address 127.29.78.190:36491)
0808 18:28:56.269377 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:56.269393 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:56.269432 (+ 39us) server_negotiation.cc:407] Connection header received
0808 18:28:56.270059 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:56.270075 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:56.270519 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:56.270861 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:56.271303 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:56.272368 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:56.273273 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:56.273743 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:56.273946 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:56.274811 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:56.274817 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:56.274829 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:28:56.275210 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:56.275266 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:28:56.275286 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:56.275396 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:56.275561 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 18:28:56.275809 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":136,"threads_started":1}
I20260808 18:28:56.277631 3478 raft_consensus.cc:348] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceaaf42868514b20a3aef4fe58107cc0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36491 } }
I20260808 18:28:56.278007 3478 raft_consensus.cc:374] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:56.278107 3478 raft_consensus.cc:729] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceaaf42868514b20a3aef4fe58107cc0, State: Initialized, Role: FOLLOWER
I20260808 18:28:56.278801 3478 consensus_queue.cc:260] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceaaf42868514b20a3aef4fe58107cc0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36491 } }
I20260808 18:28:56.279026 3478 raft_consensus.cc:388] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:28:56.279105 3478 raft_consensus.cc:482] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:28:56.279227 3478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:56.279621 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:28:56.280242 3478 raft_consensus.cc:504] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceaaf42868514b20a3aef4fe58107cc0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36491 } }
I20260808 18:28:56.280607 3478 leader_election.cc:302] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ceaaf42868514b20a3aef4fe58107cc0; no voters:
I20260808 18:28:56.280978 3478 leader_election.cc:288] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:28:56.281072 3484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:56.281304 3484 raft_consensus.cc:686] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0 [term 1 LEADER]: Becoming Leader. State: Replica: ceaaf42868514b20a3aef4fe58107cc0, State: Running, Role: LEADER
I20260808 18:28:56.281621 3484 consensus_queue.cc:237] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceaaf42868514b20a3aef4fe58107cc0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36491 } }
I20260808 18:28:56.282197 3478 sys_catalog.cc:564] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:28:56.283355 3485 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ceaaf42868514b20a3aef4fe58107cc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceaaf42868514b20a3aef4fe58107cc0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36491 } } }
I20260808 18:28:56.283540 3485 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0 [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:56.283355 3486 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceaaf42868514b20a3aef4fe58107cc0. Latest consensus state: current_term: 1 leader_uuid: "ceaaf42868514b20a3aef4fe58107cc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceaaf42868514b20a3aef4fe58107cc0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36491 } } }
I20260808 18:28:56.283774 3486 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0 [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:56.284133 3493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:28:56.286885 3493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:28:56.292037 3493 catalog_manager.cc:1347] Generated new cluster ID: 7841133a4673454099b353b992783bd2
I20260808 18:28:56.292102 3493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:28:56.301699 3493 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:28:56.302589 3493 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:28:56.306846 3493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ceaaf42868514b20a3aef4fe58107cc0: Generated new TSK 0
I20260808 18:28:56.307562 3493 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 18:28:56.336254 3510 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260808 18:28:56.336350 3510 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 18:28:56.337203 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:56.320308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34572 (local address 127.29.78.190:36491)
0808 18:28:56.320458 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:56.320462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:56.320625 (+ 163us) server_negotiation.cc:407] Connection header received
0808 18:28:56.320832 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:56.320835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:56.320898 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:56.321012 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0808 18:28:56.321449 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:56.322487 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:56.323795 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:56.324045 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:56.324156 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:56.324481 (+ 325us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 18:28:56.336731 (+ 12250us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 18:28:56.337031 (+ 300us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34572: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":56}
W20260808 18:28:56.337265 3509 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:28:56.320220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36491 (local address 127.0.0.1:34572)
0808 18:28:56.320527 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:56.320542 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:56.320663 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:56.321148 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:56.321154 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:56.321164 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0808 18:28:56.321306 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:56.321314 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:56.322636 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:56.322640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:56.323671 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:56.323681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:56.324182 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:56.324186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:56.324281 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:56.324304 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 18:28:56.336929 (+ 12625us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 18:28:56.337093 (+ 164us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.78.190:36491: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":109,"threads_started":1}
W20260808 18:28:56.337286 3482 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34572: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260808 18:28:56.337800 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 3403
2026-08-08T18:28:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (301 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260808 18:28:56.381124 30010 mini_oidc.cc:152] Starting JWKS server
I20260808 18:28:56.381963 30010 webserver.cc:460] Webserver started at https://127.0.0.1:38529/ using document root <none> and password file <none>
I20260808 18:28:56.382046 30010 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 18:28:56.382294 30010 webserver.cc:460] Webserver started at http://127.0.0.1:34387/ using document root <none> and password file <none>
2026-08-08T18:28:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:28:56Z Disabled control of system clock
I20260808 18:28:56.392661 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:42965
--webserver_interface=127.29.78.190
--webserver_port=0
--builtin_ntp_servers=127.29.78.148:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:42965
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38529/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 18:28:56.501823 3516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:56.502067 3516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:56.502104 3516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:56.505896 3516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:28:56.505946 3516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:56.505961 3516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:56.505978 3516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:28:56.505992 3516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 18:28:56.506008 3516 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 18:28:56.506021 3516 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38529/jwks.json
I20260808 18:28:56.510673 3516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:42965
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:42965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38529/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:56.511616 3516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:56.512897 3516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:56.521271 3521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:56.521278 3522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:56.521311 3524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:56.521634 3516 server_base.cc:1034] running on GCE node
I20260808 18:28:56.522164 3516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:56.523283 3516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:56.524484 3516 hybrid_clock.cc:648] HybridClock initialized: now 1786213736524460 us; error 45 us; skew 500 ppm
I20260808 18:28:56.532785 3516 webserver.cc:460] Webserver started at http://127.29.78.190:35613/ using document root <none> and password file <none>
I20260808 18:28:56.533427 3516 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:56.533527 3516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:56.533774 3516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:56.536003 3516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "8d69d4f3c2214ef6beab402da3da9ae5"
format_stamp: "Formatted at 2026-08-08 18:28:56 on dist-test-slave-gp6n"
server_key: "c4840de2cdf2e8cd4a96b9b768ef10ea"
server_key_iv: "a2693e27f4b82c41d8b481fb41ed897a"
server_key_version: "encryptionkey@0"
I20260808 18:28:56.536587 3516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "8d69d4f3c2214ef6beab402da3da9ae5"
format_stamp: "Formatted at 2026-08-08 18:28:56 on dist-test-slave-gp6n"
server_key: "c4840de2cdf2e8cd4a96b9b768ef10ea"
server_key_iv: "a2693e27f4b82c41d8b481fb41ed897a"
server_key_version: "encryptionkey@0"
I20260808 18:28:56.540562 3516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:28:56.543308 3530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:56.544622 3516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:28:56.544775 3516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "8d69d4f3c2214ef6beab402da3da9ae5"
format_stamp: "Formatted at 2026-08-08 18:28:56 on dist-test-slave-gp6n"
server_key: "c4840de2cdf2e8cd4a96b9b768ef10ea"
server_key_iv: "a2693e27f4b82c41d8b481fb41ed897a"
server_key_version: "encryptionkey@0"
I20260808 18:28:56.544916 3516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:56.564111 3516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:56.564837 3516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:56.565057 3516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:56.573082 3516 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:42965
I20260808 18:28:56.573105 3584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:42965 every 8 connection(s)
I20260808 18:28:56.574085 3516 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:28:56.577268 3585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:56.578519 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 3516
I20260808 18:28:56.578683 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:28:56.579007 30010 external_mini_cluster.cc:1428] Setting key eeae27c8e7d8c2e760bc939d42c53ac0
I20260808 18:28:56.583700 3585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5: Bootstrap starting.
I20260808 18:28:56.586232 3585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:56.587109 3585 log.cc:824] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:56.588685 3590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:56.580465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37994 (local address 127.29.78.190:42965)
0808 18:28:56.581099 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:56.581116 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:56.581155 (+ 39us) server_negotiation.cc:407] Connection header received
0808 18:28:56.581847 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:56.581865 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:56.582327 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:56.582701 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:56.583063 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:56.584145 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:56.585368 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:56.585877 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:56.586006 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:56.586923 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:56.586928 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:56.586940 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:28:56.587371 (+ 431us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:56.587431 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:28:56.587456 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:56.587571 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:56.587729 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 18:28:56.588020 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":137,"threads_started":1}
I20260808 18:28:56.588935 3585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5: No bootstrap required, opened a new log
I20260808 18:28:56.591370 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:28:56.592660 3585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d69d4f3c2214ef6beab402da3da9ae5" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42965 } }
I20260808 18:28:56.592893 3585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:56.592943 3585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d69d4f3c2214ef6beab402da3da9ae5, State: Initialized, Role: FOLLOWER
I20260808 18:28:56.593320 3585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d69d4f3c2214ef6beab402da3da9ae5" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42965 } }
I20260808 18:28:56.593442 3585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:28:56.593488 3585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:28:56.593555 3585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:56.594317 3585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d69d4f3c2214ef6beab402da3da9ae5" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42965 } }
I20260808 18:28:56.594647 3585 leader_election.cc:302] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d69d4f3c2214ef6beab402da3da9ae5; no voters:
I20260808 18:28:56.594895 3585 leader_election.cc:288] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:28:56.595014 3592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:56.595285 3592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5 [term 1 LEADER]: Becoming Leader. State: Replica: 8d69d4f3c2214ef6beab402da3da9ae5, State: Running, Role: LEADER
I20260808 18:28:56.595597 3592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d69d4f3c2214ef6beab402da3da9ae5" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42965 } }
I20260808 18:28:56.596010 3585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:28:56.597148 3594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d69d4f3c2214ef6beab402da3da9ae5. Latest consensus state: current_term: 1 leader_uuid: "8d69d4f3c2214ef6beab402da3da9ae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d69d4f3c2214ef6beab402da3da9ae5" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42965 } } }
I20260808 18:28:56.597285 3594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5 [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:56.597636 3593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d69d4f3c2214ef6beab402da3da9ae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d69d4f3c2214ef6beab402da3da9ae5" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42965 } } }
I20260808 18:28:56.597800 3593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5 [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:56.598582 3602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:28:56.601380 3602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:28:56.606261 3602 catalog_manager.cc:1347] Generated new cluster ID: ff62cd0630d248f4be25fb3dfd38c116
I20260808 18:28:56.606321 3602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:28:56.620209 3602 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:28:56.621030 3602 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:28:56.633056 3602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d69d4f3c2214ef6beab402da3da9ae5: Generated new TSK 0
I20260808 18:28:56.633675 3602 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:28:56.649140 3590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:56.644088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38008 (local address 127.29.78.190:42965)
0808 18:28:56.644242 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:56.644247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:56.644397 (+ 150us) server_negotiation.cc:407] Connection header received
0808 18:28:56.644573 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:56.644576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:56.644623 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:56.644714 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0808 18:28:56.644980 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:56.645518 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:56.646805 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:56.647019 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:56.647099 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:56.647410 (+ 311us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 18:28:56.648777 (+ 1367us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38529/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 18:28:56.648999 (+ 222us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38008: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38529/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":66}
W20260808 18:28:56.649206 3590 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38008: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38529/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260808 18:28:56.651067 3620 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:28:56.643986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42965 (local address 127.0.0.1:38008)
0808 18:28:56.644299 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:56.644313 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:56.644440 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:56.644736 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:56.644739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:56.644747 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0808 18:28:56.644867 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:56.644873 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:56.645641 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:56.645645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:56.646684 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:56.646696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:56.647135 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:56.647139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:56.647228 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:56.647256 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 18:28:56.648935 (+ 1679us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38529/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 18:28:56.650887 (+ 1952us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.78.190:42965: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38529/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":1760,"thread_start_us":118,"threads_started":1}
I20260808 18:28:56.651590 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 3516
E20260808 18:28:56.664781 3621 webserver.cc:533] Webserver: sslize failed:
E20260808 18:28:56.664845 3621 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-08T18:28:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (314 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-08T18:28:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:28:56Z Disabled control of system clock
I20260808 18:28:56.704618 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:36591
--webserver_interface=127.29.78.190
--webserver_port=0
--builtin_ntp_servers=127.29.78.148:40707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:36591
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786213708769829-30010-0/testchainca.cert with env {}
W20260808 18:28:56.815231 3626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:56.815487 3626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:56.815526 3626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:56.819519 3626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:28:56.819581 3626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:56.819595 3626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:56.819612 3626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:28:56.819628 3626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 18:28:56.819644 3626 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 18:28:56.819658 3626 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260808 18:28:56.824863 3626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:40707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:36591
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:36591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786213708769829-30010-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:56.825848 3626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:56.826905 3626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:56.835515 3632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:56.835552 3635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:56.835614 3626 server_base.cc:1034] running on GCE node
W20260808 18:28:56.835563 3633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:56.836226 3626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:56.837328 3626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:56.838528 3626 hybrid_clock.cc:648] HybridClock initialized: now 1786213736838506 us; error 25 us; skew 500 ppm
I20260808 18:28:56.840677 3626 webserver.cc:460] Webserver started at http://127.29.78.190:38957/ using document root <none> and password file <none>
I20260808 18:28:56.841267 3626 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:56.841364 3626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:56.841611 3626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:56.843752 3626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "47cd028e0f614d76ae5736382b64b90b"
format_stamp: "Formatted at 2026-08-08 18:28:56 on dist-test-slave-gp6n"
server_key: "dfffe689663d9996a33230365ba2069a"
server_key_iv: "d71ed7887fa03d36fc7fc9316a7444b2"
server_key_version: "encryptionkey@0"
I20260808 18:28:56.844334 3626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "47cd028e0f614d76ae5736382b64b90b"
format_stamp: "Formatted at 2026-08-08 18:28:56 on dist-test-slave-gp6n"
server_key: "dfffe689663d9996a33230365ba2069a"
server_key_iv: "d71ed7887fa03d36fc7fc9316a7444b2"
server_key_version: "encryptionkey@0"
I20260808 18:28:56.848214 3626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 18:28:56.850800 3643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:56.852006 3626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:28:56.852147 3626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "47cd028e0f614d76ae5736382b64b90b"
format_stamp: "Formatted at 2026-08-08 18:28:56 on dist-test-slave-gp6n"
server_key: "dfffe689663d9996a33230365ba2069a"
server_key_iv: "d71ed7887fa03d36fc7fc9316a7444b2"
server_key_version: "encryptionkey@0"
I20260808 18:28:56.852263 3626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:56.875511 3626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:56.876204 3626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:56.876406 3626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:56.883522 3626 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:36591
I20260808 18:28:56.883539 3697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:36591 every 8 connection(s)
I20260808 18:28:56.884431 3626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:28:56.887351 3698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:56.890636 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 3626
I20260808 18:28:56.890755 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:28:56.891052 30010 external_mini_cluster.cc:1428] Setting key f5d5cca34c17b3bc89181a1c71882cb0
I20260808 18:28:56.893126 3698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b: Bootstrap starting.
I20260808 18:28:56.895306 3698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:56.896018 3698 log.cc:824] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:56.897895 3698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b: No bootstrap required, opened a new log
I20260808 18:28:56.900032 3702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:56.892337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58172 (local address 127.29.78.190:36591)
0808 18:28:56.892965 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:56.892980 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:56.893014 (+ 34us) server_negotiation.cc:407] Connection header received
0808 18:28:56.893686 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:56.893704 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:56.894078 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:56.894427 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:56.894850 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:56.895924 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:56.896874 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:56.897334 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:56.897518 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:56.898381 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:56.898392 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:56.898443 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:28:56.898811 (+ 368us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:56.898864 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:28:56.898884 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:56.898990 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:56.899141 (+ 151us) server_negotiation.cc:299] Negotiation successful
0808 18:28:56.899380 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":122,"threads_started":1}
I20260808 18:28:56.901574 3698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd028e0f614d76ae5736382b64b90b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36591 } }
I20260808 18:28:56.901851 3698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:56.901926 3698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47cd028e0f614d76ae5736382b64b90b, State: Initialized, Role: FOLLOWER
I20260808 18:28:56.902336 3698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd028e0f614d76ae5736382b64b90b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36591 } }
I20260808 18:28:56.902527 3698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:28:56.902594 3698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:28:56.902686 3698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:56.902915 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:28:56.903597 3698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd028e0f614d76ae5736382b64b90b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36591 } }
I20260808 18:28:56.903973 3698 leader_election.cc:302] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47cd028e0f614d76ae5736382b64b90b; no voters:
I20260808 18:28:56.904276 3698 leader_election.cc:288] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:28:56.904381 3704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:56.904604 3704 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b [term 1 LEADER]: Becoming Leader. State: Replica: 47cd028e0f614d76ae5736382b64b90b, State: Running, Role: LEADER
I20260808 18:28:56.904881 3704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd028e0f614d76ae5736382b64b90b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36591 } }
I20260808 18:28:56.905428 3698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:28:56.905905 3704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47cd028e0f614d76ae5736382b64b90b. Latest consensus state: current_term: 1 leader_uuid: "47cd028e0f614d76ae5736382b64b90b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd028e0f614d76ae5736382b64b90b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36591 } } }
I20260808 18:28:56.906029 3704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:56.906349 3705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47cd028e0f614d76ae5736382b64b90b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd028e0f614d76ae5736382b64b90b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36591 } } }
I20260808 18:28:56.906498 3705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:56.906450 3709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:28:56.909425 3709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:28:56.915633 3709 catalog_manager.cc:1347] Generated new cluster ID: a8a2de0dc17c4945ba0f5a14ed8bc1bd
I20260808 18:28:56.915700 3709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:28:56.931037 3709 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:28:56.932365 3709 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:28:56.939294 3709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47cd028e0f614d76ae5736382b64b90b: Generated new TSK 0
I20260808 18:28:56.940143 3709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:28:57.044148 3702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:56.951465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58186 (local address 127.29.78.190:36591)
0808 18:28:56.951597 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:56.951601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:56.951780 (+ 179us) server_negotiation.cc:407] Connection header received
0808 18:28:56.951907 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:56.951911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:56.951963 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:56.952079 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0808 18:28:56.952282 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:56.953128 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:56.954062 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:56.954441 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:56.954544 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:56.954864 (+ 320us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 18:28:57.043653 (+ 88789us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 18:28:57.043985 (+ 332us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58186: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":49}
W20260808 18:28:57.044250 3702 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58186: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260808 18:28:57.044672 3729 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:28:56.951387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36591 (local address 127.0.0.1:58186)
0808 18:28:56.951699 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:56.951712 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:56.951805 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:56.952080 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:56.952082 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:56.952088 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0808 18:28:56.952183 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:56.952189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:56.953266 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:56.953270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:56.953941 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:56.953951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:56.954567 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:56.954570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:56.954643 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:56.954667 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 18:28:57.043925 (+ 89258us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 18:28:57.044148 (+ 223us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.78.190:36591: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":131,"threads_started":1}
I20260808 18:28:57.045420 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 3626
2026-08-08T18:28:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (372 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (129 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (125 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:28:57 dist-test-slave-gp6n krb5kdc[3744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:28:57 dist-test-slave-gp6n krb5kdc[3744](info): set up 2 sockets
Aug 08 18:28:57 dist-test-slave-gp6n krb5kdc[3744](info): commencing operation
krb5kdc: starting...
W20260808 18:28:58.470541 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:28:58 dist-test-slave-gp6n krb5kdc[3744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:28:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:28:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:28:58.602130 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:39497
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:43851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:39497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:58.712976 3759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:58.713256 3759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:58.713338 3759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:58.717280 3759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:28:58.717366 3759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:58.717406 3759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:58.717465 3759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:28:58.717504 3759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:28:58.722730 3759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:43851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:39497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:39497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:58.723798 3759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:58.724785 3759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:58.733453 3767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:58.733398 3764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:58.733566 3759 server_base.cc:1034] running on GCE node
W20260808 18:28:58.733417 3765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:58.734249 3759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:58.735368 3759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:58.736563 3759 hybrid_clock.cc:648] HybridClock initialized: now 1786213738736551 us; error 24 us; skew 500 ppm
Aug 08 18:28:58 dist-test-slave-gp6n krb5kdc[3744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213738, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:58.739459 3759 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:28:58.740253 3759 webserver.cc:460] Webserver started at http://127.29.78.190:43781/ using document root <none> and password file <none>
I20260808 18:28:58.740760 3759 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:58.740818 3759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:58.740979 3759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:58.742996 3759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "eed9a9f32a27472cb3cf55d7093de5ff"
format_stamp: "Formatted at 2026-08-08 18:28:58 on dist-test-slave-gp6n"
server_key: "9011368892dc791db2385e444a5f03a2"
server_key_iv: "d4d1687bf9b0ae104cf94359b1a14e5d"
server_key_version: "encryptionkey@0"
I20260808 18:28:58.743510 3759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "eed9a9f32a27472cb3cf55d7093de5ff"
format_stamp: "Formatted at 2026-08-08 18:28:58 on dist-test-slave-gp6n"
server_key: "9011368892dc791db2385e444a5f03a2"
server_key_iv: "d4d1687bf9b0ae104cf94359b1a14e5d"
server_key_version: "encryptionkey@0"
I20260808 18:28:58.747139 3759 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 18:28:58.749439 3775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:58.750798 3759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:28:58.750901 3759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "eed9a9f32a27472cb3cf55d7093de5ff"
format_stamp: "Formatted at 2026-08-08 18:28:58 on dist-test-slave-gp6n"
server_key: "9011368892dc791db2385e444a5f03a2"
server_key_iv: "d4d1687bf9b0ae104cf94359b1a14e5d"
server_key_version: "encryptionkey@0"
I20260808 18:28:58.750989 3759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:58.762006 3759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:58.765206 3759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:58.765352 3759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:58.772292 3759 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:39497
I20260808 18:28:58.772405 3830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:39497 every 8 connection(s)
I20260808 18:28:58.773272 3759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:28:58.776058 3831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:28:58.778725 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 3759
I20260808 18:28:58.778852 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:28:58.779150 30010 external_mini_cluster.cc:1428] Setting key ba3b1ca2b8f653379812746e60752988
I20260808 18:28:58.782927 3831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff: Bootstrap starting.
I20260808 18:28:58.785495 3831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff: Neither blocks nor log segments found. Creating new log.
I20260808 18:28:58.786301 3831 log.cc:824] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff: Log is configured to *not* fsync() on all Append() calls
I20260808 18:28:58.788249 3831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff: No bootstrap required, opened a new log
Aug 08 18:28:58 dist-test-slave-gp6n krb5kdc[3744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:58.792500 3831 raft_consensus.cc:348] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed9a9f32a27472cb3cf55d7093de5ff" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39497 } }
I20260808 18:28:58.792799 3831 raft_consensus.cc:374] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:28:58.792877 3831 raft_consensus.cc:729] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed9a9f32a27472cb3cf55d7093de5ff, State: Initialized, Role: FOLLOWER
I20260808 18:28:58.793341 3831 consensus_queue.cc:260] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed9a9f32a27472cb3cf55d7093de5ff" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39497 } }
I20260808 18:28:58.793488 3831 raft_consensus.cc:388] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:28:58.793563 3831 raft_consensus.cc:482] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:28:58.793672 3831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:28:58.794691 3831 raft_consensus.cc:504] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed9a9f32a27472cb3cf55d7093de5ff" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39497 } }
I20260808 18:28:58.795017 3831 leader_election.cc:302] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eed9a9f32a27472cb3cf55d7093de5ff; no voters:
I20260808 18:28:58.795136 3834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:58.780597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35020 (local address 127.29.78.190:39497)
0808 18:28:58.781036 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:58.781054 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:58.781099 (+ 45us) server_negotiation.cc:407] Connection header received
0808 18:28:58.781920 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:58.781944 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:58.782343 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:58.782749 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:58.783572 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:58.785077 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:58.786121 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:58.786832 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:58.786950 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:58.789845 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:58.789852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:58.789867 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:58.789897 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:58.792034 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:58.792468 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:58.792474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:58.792480 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:58.792661 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:58.793071 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:58.793074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:58.793075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:58.793441 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:58.793690 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:58.793987 (+ 297us) server_negotiation.cc:299] Negotiation successful
0808 18:28:58.794294 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":123,"threads_started":1}
I20260808 18:28:58.795325 3831 leader_election.cc:288] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:28:58.795574 3836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:28:58.795758 3836 raft_consensus.cc:686] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff [term 1 LEADER]: Becoming Leader. State: Replica: eed9a9f32a27472cb3cf55d7093de5ff, State: Running, Role: LEADER
I20260808 18:28:58.796031 3836 consensus_queue.cc:237] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed9a9f32a27472cb3cf55d7093de5ff" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39497 } }
I20260808 18:28:58.796610 3831 sys_catalog.cc:564] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:28:58.797164 3836 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader eed9a9f32a27472cb3cf55d7093de5ff. Latest consensus state: current_term: 1 leader_uuid: "eed9a9f32a27472cb3cf55d7093de5ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed9a9f32a27472cb3cf55d7093de5ff" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39497 } } }
I20260808 18:28:58.797276 3836 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff [sys.catalog]: This master's current role is: LEADER
I20260808 18:28:58.797588 3837 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eed9a9f32a27472cb3cf55d7093de5ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed9a9f32a27472cb3cf55d7093de5ff" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39497 } } }
I20260808 18:28:58.797689 3837 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff [sys.catalog]: This master's current role is: LEADER
W20260808 18:28:58.800987 3852 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:28:58.801070 3852 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:28:58.801147 3842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:28:58.801879 3842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:28:58.807279 3842 catalog_manager.cc:1347] Generated new cluster ID: 6162afafc5fe4170b2f3fdb1b0221911
I20260808 18:28:58.807343 3842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:28:58.826445 3842 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:28:58.827726 3842 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:28:58.847832 3842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eed9a9f32a27472cb3cf55d7093de5ff: Generated new TSK 0
I20260808 18:28:58.848613 3842 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260808 18:28:58.881541 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39497
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:43851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:58.993799 3858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:58.994046 3858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:58.994088 3858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:58.998545 3858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:58.998602 3858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:58.998703 3858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:28:59.003883 3858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:43851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:59.004859 3858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:59.005810 3858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:59.015596 3866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:59.015594 3864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:59.015627 3863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:59.015666 3858 server_base.cc:1034] running on GCE node
I20260808 18:28:59.016311 3858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:59.016973 3858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:59.018158 3858 hybrid_clock.cc:648] HybridClock initialized: now 1786213739018127 us; error 49 us; skew 500 ppm
Aug 08 18:28:59 dist-test-slave-gp6n krb5kdc[3744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213739, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:59.020972 3858 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:28:59.021852 3858 webserver.cc:460] Webserver started at http://127.29.78.129:40435/ using document root <none> and password file <none>
I20260808 18:28:59.022472 3858 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:59.022565 3858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:59.022801 3858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:59.024821 3858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "c1e97b59ec8b4f13b68165628848249e"
format_stamp: "Formatted at 2026-08-08 18:28:59 on dist-test-slave-gp6n"
server_key: "79d03b2d172c215a84755f4c2f6e007c"
server_key_iv: "a26110ccc20141d71394dad780cc6613"
server_key_version: "encryptionkey@0"
I20260808 18:28:59.025394 3858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "c1e97b59ec8b4f13b68165628848249e"
format_stamp: "Formatted at 2026-08-08 18:28:59 on dist-test-slave-gp6n"
server_key: "79d03b2d172c215a84755f4c2f6e007c"
server_key_iv: "a26110ccc20141d71394dad780cc6613"
server_key_version: "encryptionkey@0"
I20260808 18:28:59.029135 3858 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 18:28:59.031741 3875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:59.033079 3858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:28:59.033330 3858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "c1e97b59ec8b4f13b68165628848249e"
format_stamp: "Formatted at 2026-08-08 18:28:59 on dist-test-slave-gp6n"
server_key: "79d03b2d172c215a84755f4c2f6e007c"
server_key_iv: "a26110ccc20141d71394dad780cc6613"
server_key_version: "encryptionkey@0"
I20260808 18:28:59.033454 3858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:59.072707 3858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:59.075856 3858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:59.076072 3858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:59.076690 3858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:59.077538 3858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:59.077607 3858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:59.077680 3858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:59.077730 3858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:59.086691 3858 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:44071
I20260808 18:28:59.086719 3997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:44071 every 8 connection(s)
I20260808 18:28:59.087569 3858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:28:59.088569 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 3858
I20260808 18:28:59.088690 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:28:59.088994 30010 external_mini_cluster.cc:1428] Setting key 53fa11073d060b70ae5f756605442a56
Aug 08 18:28:59 dist-test-slave-gp6n krb5kdc[3744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213739, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
I20260808 18:28:59.104054 3834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:59.089531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:32847 (local address 127.29.78.190:39497)
0808 18:28:59.089674 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:59.089678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:59.090648 (+ 970us) server_negotiation.cc:407] Connection header received
0808 18:28:59.092027 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:59.092031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:59.092087 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:59.092193 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:59.093893 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:59.094699 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:59.096826 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:59.097069 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:59.097156 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:59.100298 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:59.100301 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:59.100303 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:59.100327 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:59.101834 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:59.102461 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:59.102465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:59.102467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:59.102554 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:59.102909 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:59.102912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:59.102913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:59.103098 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:59.103205 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:59.103715 (+ 510us) server_negotiation.cc:299] Negotiation successful
0808 18:28:59.103862 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 18:28:59.105077 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:59.089948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:39497 (local address 127.29.78.129:32847)
0808 18:28:59.090485 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:59.090547 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:59.091821 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:59.092387 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:59.092400 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:59.092964 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:59.093699 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:59.093724 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:59.094861 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:59.094869 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:59.096688 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:59.096705 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:59.097259 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:59.097267 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:59.097551 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:59.098283 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:59.098312 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:59.100125 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:59.101976 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:59.101984 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:59.102000 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:59.102331 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:59.102657 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:59.102660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:59.102662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:59.102801 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:59.103205 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:59.103213 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:59.103474 (+ 261us) client_negotiation.cc:769] Sending connection context
0808 18:28:59.103659 (+ 185us) client_negotiation.cc:240] Negotiation successful
0808 18:28:59.103990 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":106,"threads_started":1}
I20260808 18:28:59.106557 3998 heartbeater.cc:347] Connected to a master server at 127.29.78.190:39497
I20260808 18:28:59.106845 3998 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:59.107491 3998 heartbeater.cc:511] Master 127.29.78.190:39497 requested a full tablet report, sending...
I20260808 18:28:59.109095 3795 ts_manager.cc:194] Registered new tserver with Master: c1e97b59ec8b4f13b68165628848249e (127.29.78.129:44071)
I20260808 18:28:59.110283 3795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:32847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260808 18:28:59.122123 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39497
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:43851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:59.231830 4004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:59.232076 4004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:59.232117 4004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:59.235987 4004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:59.236035 4004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:59.236089 4004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:28:59.241158 4004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:43851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:59.242040 4004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:59.243119 4004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:59.252607 4011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:59.252684 4004 server_base.cc:1034] running on GCE node
W20260808 18:28:59.252628 4014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:59.252629 4012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:59.253314 4004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:59.253961 4004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:59.255115 4004 hybrid_clock.cc:648] HybridClock initialized: now 1786213739255077 us; error 50 us; skew 500 ppm
Aug 08 18:28:59 dist-test-slave-gp6n krb5kdc[3744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213739, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:59.257966 4004 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:28:59.258894 4004 webserver.cc:460] Webserver started at http://127.29.78.130:37651/ using document root <none> and password file <none>
I20260808 18:28:59.259469 4004 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:59.259533 4004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:59.259779 4004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:59.261726 4004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "1e5051751e5240c4854813562e97785f"
format_stamp: "Formatted at 2026-08-08 18:28:59 on dist-test-slave-gp6n"
server_key: "1b5b02ef52194adcb607555ee5024c81"
server_key_iv: "eacc93b79e307c42623574e11741eb4f"
server_key_version: "encryptionkey@0"
I20260808 18:28:59.262288 4004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "1e5051751e5240c4854813562e97785f"
format_stamp: "Formatted at 2026-08-08 18:28:59 on dist-test-slave-gp6n"
server_key: "1b5b02ef52194adcb607555ee5024c81"
server_key_iv: "eacc93b79e307c42623574e11741eb4f"
server_key_version: "encryptionkey@0"
I20260808 18:28:59.265894 4004 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:28:59.268404 4022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:59.269850 4004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:28:59.269975 4004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "1e5051751e5240c4854813562e97785f"
format_stamp: "Formatted at 2026-08-08 18:28:59 on dist-test-slave-gp6n"
server_key: "1b5b02ef52194adcb607555ee5024c81"
server_key_iv: "eacc93b79e307c42623574e11741eb4f"
server_key_version: "encryptionkey@0"
I20260808 18:28:59.270079 4004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:59.281114 4004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:59.284458 4004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:59.284663 4004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:59.285255 4004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:59.286093 4004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:59.286149 4004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:59.286211 4004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:59.286245 4004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:59.295737 4004 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:33305
I20260808 18:28:59.295810 4146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:33305 every 8 connection(s)
I20260808 18:28:59.296795 4004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:28:59.299033 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 4004
I20260808 18:28:59.299151 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:28:59.299495 30010 external_mini_cluster.cc:1428] Setting key 317128c5783360f69c2d7f74cf2866ab
Aug 08 18:28:59 dist-test-slave-gp6n krb5kdc[3744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213739, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260808 18:28:59.313473 3834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:59.299366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:56403 (local address 127.29.78.190:39497)
0808 18:28:59.299718 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:59.299721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:59.301058 (+ 1337us) server_negotiation.cc:407] Connection header received
0808 18:28:59.303177 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:59.303180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:59.303227 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:59.303359 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:59.304582 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:59.305455 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:59.306736 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:59.307171 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:59.307306 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:59.310008 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:59.310014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:59.310019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:59.310055 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:59.311654 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:59.312117 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:59.312120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:59.312121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:59.312199 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:59.312501 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:59.312503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:59.312504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:59.312673 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:59.312748 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:59.313223 (+ 475us) server_negotiation.cc:299] Negotiation successful
0808 18:28:59.313328 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
I20260808 18:28:59.314589 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:59.299911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:39497 (local address 127.29.78.130:56403)
0808 18:28:59.300912 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:59.300958 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:59.302934 (+ 1976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:59.303568 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:59.303576 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:59.303944 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:59.304432 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:59.304447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:59.305586 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:59.305588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:59.306615 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:59.306624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:59.307319 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:59.307322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:59.307494 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:59.308102 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:59.308118 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:59.309810 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:59.311763 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:59.311768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:59.311779 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:59.312035 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:59.312281 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:59.312283 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:59.312285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:59.312421 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:59.312760 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:59.312764 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:59.312989 (+ 225us) client_negotiation.cc:769] Sending connection context
0808 18:28:59.313141 (+ 152us) client_negotiation.cc:240] Negotiation successful
0808 18:28:59.313445 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":136,"threads_started":1}
I20260808 18:28:59.315759 4148 heartbeater.cc:347] Connected to a master server at 127.29.78.190:39497
I20260808 18:28:59.316020 4148 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:59.316496 4148 heartbeater.cc:511] Master 127.29.78.190:39497 requested a full tablet report, sending...
I20260808 18:28:59.317528 3795 ts_manager.cc:194] Registered new tserver with Master: 1e5051751e5240c4854813562e97785f (127.29.78.130:33305)
I20260808 18:28:59.318197 3795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:56403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260808 18:28:59.329873 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39497
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:43851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:28:59.443085 4154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:28:59.443374 4154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:28:59.443446 4154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:28:59.447499 4154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:28:59.447576 4154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:28:59.447664 4154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:28:59.452708 4154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:43851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:28:59.453706 4154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:28:59.454799 4154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:28:59.464480 4159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:28:59.464648 4160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:59.464654 4154 server_base.cc:1034] running on GCE node
W20260808 18:28:59.464480 4163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:28:59.465240 4154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:28:59.465837 4154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:28:59.466992 4154 hybrid_clock.cc:648] HybridClock initialized: now 1786213739466955 us; error 50 us; skew 500 ppm
Aug 08 18:28:59 dist-test-slave-gp6n krb5kdc[3744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213739, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:59.469796 4154 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:28:59.470691 4154 webserver.cc:460] Webserver started at http://127.29.78.131:32807/ using document root <none> and password file <none>
I20260808 18:28:59.471201 4154 fs_manager.cc:359] Metadata directory not provided
I20260808 18:28:59.471259 4154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:28:59.471421 4154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:28:59.473408 4154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "7815af8b71434352a5f38eae81052f74"
format_stamp: "Formatted at 2026-08-08 18:28:59 on dist-test-slave-gp6n"
server_key: "1db841bf6ea0446a4edd754f1c1e4dff"
server_key_iv: "cbec5e6001889f204ad45893abed531b"
server_key_version: "encryptionkey@0"
I20260808 18:28:59.473932 4154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "7815af8b71434352a5f38eae81052f74"
format_stamp: "Formatted at 2026-08-08 18:28:59 on dist-test-slave-gp6n"
server_key: "1db841bf6ea0446a4edd754f1c1e4dff"
server_key_iv: "cbec5e6001889f204ad45893abed531b"
server_key_version: "encryptionkey@0"
I20260808 18:28:59.477572 4154 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 18:28:59.480023 4170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:59.481141 4154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:28:59.481256 4154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "7815af8b71434352a5f38eae81052f74"
format_stamp: "Formatted at 2026-08-08 18:28:59 on dist-test-slave-gp6n"
server_key: "1db841bf6ea0446a4edd754f1c1e4dff"
server_key_iv: "cbec5e6001889f204ad45893abed531b"
server_key_version: "encryptionkey@0"
I20260808 18:28:59.481357 4154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:28:59.514581 4154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:28:59.517899 4154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:28:59.518103 4154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:28:59.518724 4154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:28:59.519557 4154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:28:59.519629 4154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:59.519701 4154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:28:59.519755 4154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:28:59.528782 4154 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:44209
I20260808 18:28:59.528816 4295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:44209 every 8 connection(s)
I20260808 18:28:59.529696 4154 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:28:59.537925 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 4154
I20260808 18:28:59.538072 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:28:59.538360 30010 external_mini_cluster.cc:1428] Setting key 37926b95448a6e4064f75f65363467d5
Aug 08 18:28:59 dist-test-slave-gp6n krb5kdc[3744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213739, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:28:59.543733 3834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:59.531531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:59987 (local address 127.29.78.190:39497)
0808 18:28:59.531680 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:59.531685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:59.532566 (+ 881us) server_negotiation.cc:407] Connection header received
0808 18:28:59.534013 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:59.534016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:59.534063 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:59.534138 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:59.535397 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:59.535937 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:59.537245 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:59.537555 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:59.537637 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:59.540299 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:59.540302 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:59.540304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:59.540328 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:59.541659 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:59.542176 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:59.542184 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:59.542185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:59.542262 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:59.542654 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:59.542659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:59.542660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:59.542841 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:59.542953 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:59.543451 (+ 498us) server_negotiation.cc:299] Negotiation successful
0808 18:28:59.543569 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 18:28:59.544476 4299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:59.531949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:39497 (local address 127.29.78.131:59987)
0808 18:28:59.532407 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0808 18:28:59.532454 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 18:28:59.533809 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:28:59.534307 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:28:59.534314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:28:59.534765 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:28:59.535267 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:59.535282 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:59.536056 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:59.536059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:59.537126 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:28:59.537136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:28:59.537693 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:28:59.537696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:28:59.537866 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:59.538648 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:28:59.538669 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:28:59.540127 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:28:59.541776 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:59.541782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:59.541796 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:59.542063 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:59.542364 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:28:59.542367 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:28:59.542369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:28:59.542554 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:28:59.542954 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:28:59.542961 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:28:59.543202 (+ 241us) client_negotiation.cc:769] Sending connection context
0808 18:28:59.543382 (+ 180us) client_negotiation.cc:240] Negotiation successful
0808 18:28:59.543692 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":113,"threads_started":1}
I20260808 18:28:59.545531 4296 heartbeater.cc:347] Connected to a master server at 127.29.78.190:39497
I20260808 18:28:59.545782 4296 heartbeater.cc:464] Registering TS with master...
I20260808 18:28:59.546272 4296 heartbeater.cc:511] Master 127.29.78.190:39497 requested a full tablet report, sending...
I20260808 18:28:59.547318 3795 ts_manager.cc:194] Registered new tserver with Master: 7815af8b71434352a5f38eae81052f74 (127.29.78.131:44209)
I20260808 18:28:59.548028 3795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:59987
I20260808 18:28:59.552162 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/test-admin.keytab.
Aug 08 18:28:59 dist-test-slave-gp6n krb5kdc[3744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:59.571295 30010 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260808 18:28:59.574069 4304 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:28:59.572587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37318 (local address 127.29.78.129:44071)
0808 18:28:59.572872 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:59.572888 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:59.572925 (+ 37us) server_negotiation.cc:407] Connection header received
0808 18:28:59.572975 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:59.572980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:59.573265 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:59.573390 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:59.573869 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37318: BlockingRecv error: recv got EOF from 127.0.0.1:37318 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":69,"threads_started":1}
W20260808 18:28:59.574144 30010 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/krb5cc' of type FILE
Aug 08 18:28:59 dist-test-slave-gp6n krb5kdc[3744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:59.575306 30010 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 18:28:59 dist-test-slave-gp6n krb5kdc[3744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:59.576320 30010 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 18:28:59 dist-test-slave-gp6n krb5kdc[3744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260808 18:28:59.586211 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:59.577184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37330 (local address 127.29.78.129:44071)
0808 18:28:59.577318 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:59.577324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:59.577416 (+ 92us) server_negotiation.cc:407] Connection header received
0808 18:28:59.577571 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:59.577574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:59.577624 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:59.577712 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:59.578247 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:59.579110 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:59.579958 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:59.580341 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:59.580460 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:59.582832 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:59.582842 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:59.582854 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:59.582878 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:59.584315 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:59.584758 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:59.584764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:59.584765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:59.584888 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:59.585186 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:59.585190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:59.585191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:59.585434 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:59.585651 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:59.585926 (+ 275us) server_negotiation.cc:299] Negotiation successful
0808 18:28:59.586078 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 18:28:59.590034 4304 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:28:59.589033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37336 (local address 127.29.78.129:44071)
0808 18:28:59.589144 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:59.589148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:59.589276 (+ 128us) server_negotiation.cc:407] Connection header received
0808 18:28:59.589361 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:59.589363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:59.589396 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:59.589483 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:59.589922 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37336: BlockingRecv error: recv got EOF from 127.0.0.1:37336 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260808 18:28:59.590173 30010 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/krb5cc' of type FILE
Aug 08 18:28:59 dist-test-slave-gp6n krb5kdc[3744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:59.591195 30010 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 18:28:59 dist-test-slave-gp6n krb5kdc[3744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:59.592057 30010 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 18:28:59 dist-test-slave-gp6n krb5kdc[3744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260808 18:28:59.601006 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:59.592863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37352 (local address 127.29.78.129:44071)
0808 18:28:59.592988 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:59.592992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:59.593064 (+ 72us) server_negotiation.cc:407] Connection header received
0808 18:28:59.593150 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:59.593152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:59.593183 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:59.593245 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:59.593823 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:59.594295 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:59.595259 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:59.595523 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:59.595607 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:59.597929 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:59.597932 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:59.597934 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:59.597955 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:59.599438 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:59.599837 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:59.599839 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:59.599841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:59.599910 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:59.600258 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:59.600260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:59.600261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:59.600405 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:59.600515 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:59.600708 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 18:28:59.600823 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 18:28:59.603456 4304 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:28:59.602653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.29.78.129:44071)
0808 18:28:59.602776 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:59.602780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:59.602830 (+ 50us) server_negotiation.cc:407] Connection header received
0808 18:28:59.602932 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:59.602934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:59.602966 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:59.603029 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:59.603369 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37356: BlockingRecv error: recv got EOF from 127.0.0.1:37356 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260808 18:28:59.603560 30010 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786213708769829-30010-0/krb5kdc/krb5cc' of type FILE
Aug 08 18:28:59 dist-test-slave-gp6n krb5kdc[3744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:59.604525 30010 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 18:28:59 dist-test-slave-gp6n krb5kdc[3744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:28:59.605422 30010 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 18:28:59 dist-test-slave-gp6n krb5kdc[3744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260808 18:28:59.613394 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:28:59.606247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.29.78.129:44071)
0808 18:28:59.606352 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0808 18:28:59.606355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:28:59.606541 (+ 186us) server_negotiation.cc:407] Connection header received
0808 18:28:59.606626 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:28:59.606628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:28:59.606660 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:28:59.606715 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:28:59.607124 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:59.607590 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:59.608313 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:28:59.608517 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:28:59.608590 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:28:59.610746 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:28:59.610749 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:28:59.610751 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:28:59.610770 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:28:59.612121 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:59.612483 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:59.612487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:59.612488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:59.612571 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:28:59.612803 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:28:59.612806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:28:59.612807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:28:59.612947 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:28:59.613048 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 18:28:59.613223 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 18:28:59.613302 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 18:28:59.614387 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 3858
I20260808 18:28:59.622212 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 4004
I20260808 18:28:59.628463 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 4154
I20260808 18:28:59.638125 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 3759
2026-08-08T18:28:59Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2338 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:28:59 dist-test-slave-gp6n krb5kdc[4317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:28:59 dist-test-slave-gp6n krb5kdc[4317](info): set up 2 sockets
Aug 08 18:28:59 dist-test-slave-gp6n krb5kdc[4317](info): commencing operation
krb5kdc: starting...
W20260808 18:29:00.774564 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.003s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:29:00 dist-test-slave-gp6n krb5kdc[4317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:29:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:29:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.190.keytab.
Entry for principal oryx/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.190.keytab.
I20260808 18:29:00.905758 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:34937
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.190.keytab
--principal=oryx/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:34937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:29:01.019357 4332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:01.019666 4332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:01.019711 4332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:01.024246 4332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:29:01.024331 4332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:01.024374 4332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:01.024402 4332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:29:01.024436 4332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:29:01.034040 4332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:34937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.190.keytab
--principal=oryx/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:34937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:01.035605 4332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:01.037101 4332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:01.048734 4338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:01.048926 4332 server_base.cc:1034] running on GCE node
W20260808 18:29:01.048728 4340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:01.049208 4337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:01.049911 4332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:01.051184 4332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:01.052459 4332 hybrid_clock.cc:648] HybridClock initialized: now 1786213741052437 us; error 43 us; skew 500 ppm
Aug 08 18:29:01 dist-test-slave-gp6n krb5kdc[4317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213741, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:01.056074 4332 init.cc:377] Logged in from keytab as oryx/127.29.78.190@KRBTEST.COM (short username oryx)
I20260808 18:29:01.057279 4332 webserver.cc:460] Webserver started at http://127.29.78.190:38409/ using document root <none> and password file <none>
I20260808 18:29:01.057994 4332 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:01.058060 4332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:01.058318 4332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:01.061494 4332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "181bd4194326418da33169a4f27346f8"
format_stamp: "Formatted at 2026-08-08 18:29:01 on dist-test-slave-gp6n"
server_key: "041605deffd2add64fc6928bf3ec3f0d"
server_key_iv: "3397a1e7e29daf9371d83a27bab00457"
server_key_version: "encryptionkey@0"
I20260808 18:29:01.062283 4332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "181bd4194326418da33169a4f27346f8"
format_stamp: "Formatted at 2026-08-08 18:29:01 on dist-test-slave-gp6n"
server_key: "041605deffd2add64fc6928bf3ec3f0d"
server_key_iv: "3397a1e7e29daf9371d83a27bab00457"
server_key_version: "encryptionkey@0"
I20260808 18:29:01.067816 4332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 18:29:01.071317 4348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:01.073030 4332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:29:01.073201 4332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "181bd4194326418da33169a4f27346f8"
format_stamp: "Formatted at 2026-08-08 18:29:01 on dist-test-slave-gp6n"
server_key: "041605deffd2add64fc6928bf3ec3f0d"
server_key_iv: "3397a1e7e29daf9371d83a27bab00457"
server_key_version: "encryptionkey@0"
I20260808 18:29:01.073388 4332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:01.095315 4332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:01.098547 4332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:01.098758 4332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:01.106676 4332 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:34937
I20260808 18:29:01.106997 4405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:34937 every 8 connection(s)
I20260808 18:29:01.107812 4332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:29:01.110690 4406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:01.113485 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 4332
I20260808 18:29:01.113652 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:29:01.113958 30010 external_mini_cluster.cc:1428] Setting key 2e3c2ff4d5f887fc65ecb8a1d9c61527
I20260808 18:29:01.116787 4406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8: Bootstrap starting.
I20260808 18:29:01.119510 4406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:01.120321 4406 log.cc:824] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8: Log is configured to *not* fsync() on all Append() calls
I20260808 18:29:01.122593 4406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8: No bootstrap required, opened a new log
Aug 08 18:29:01 dist-test-slave-gp6n krb5kdc[4317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260808 18:29:01.126909 4406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } }
I20260808 18:29:01.127202 4406 raft_consensus.cc:374] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:01.127324 4406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181bd4194326418da33169a4f27346f8, State: Initialized, Role: FOLLOWER
I20260808 18:29:01.127827 4406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } }
I20260808 18:29:01.127986 4406 raft_consensus.cc:388] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:29:01.128053 4406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:29:01.128151 4406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:01.129061 4406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } }
I20260808 18:29:01.129447 4406 leader_election.cc:302] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 181bd4194326418da33169a4f27346f8; no voters:
I20260808 18:29:01.129802 4406 leader_election.cc:288] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:29:01.129870 4411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:29:01.130072 4411 raft_consensus.cc:686] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [term 1 LEADER]: Becoming Leader. State: Replica: 181bd4194326418da33169a4f27346f8, State: Running, Role: LEADER
I20260808 18:29:01.130268 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:01.115451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59888 (local address 127.29.78.190:34937)
0808 18:29:01.115894 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:01.115921 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:01.115965 (+ 44us) server_negotiation.cc:407] Connection header received
0808 18:29:01.116752 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:01.116776 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:01.117203 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:01.117552 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:01.118256 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:01.119791 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:01.120694 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:01.121289 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:01.121445 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:01.124256 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:01.124262 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:01.124280 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:01.124310 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:01.126980 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:01.127547 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:01.127552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:01.127556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:01.127676 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:01.127970 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:01.127977 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:01.127979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:01.128485 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:01.128710 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:01.129001 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 18:29:01.129339 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":119,"threads_started":1}
I20260808 18:29:01.130368 4411 consensus_queue.cc:237] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } }
I20260808 18:29:01.131125 4406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:29:01.133327 4413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 181bd4194326418da33169a4f27346f8. Latest consensus state: current_term: 1 leader_uuid: "181bd4194326418da33169a4f27346f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } }
I20260808 18:29:01.133759 4413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [sys.catalog]: This master's current role is: LEADER
I20260808 18:29:01.133551 4412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "181bd4194326418da33169a4f27346f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } }
I20260808 18:29:01.133978 4412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [sys.catalog]: This master's current role is: LEADER
W20260808 18:29:01.136308 4425 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:29:01.136394 4425 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:29:01.136494 4426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:29:01.137305 4426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:29:01.142275 4426 catalog_manager.cc:1347] Generated new cluster ID: 79b2d9f0f7a04df6b2088bd4ca8342eb
I20260808 18:29:01.142339 4426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:29:01.160857 4426 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:29:01.162190 4426 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:29:01.166486 4426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8: Generated new TSK 0
I20260808 18:29:01.167191 4426 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.129.keytab.
Entry for principal oryx/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.129.keytab.
I20260808 18:29:01.201581 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34937
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.129.keytab
--principal=oryx/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:29:01.334873 4434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:01.335151 4434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:01.335223 4434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:01.339191 4434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:01.339275 4434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:01.339368 4434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:29:01.345716 4434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.129.keytab
--principal=oryx/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:01.346884 4434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:01.347895 4434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:29:01.357890 4434 server_base.cc:1034] running on GCE node
W20260808 18:29:01.357862 4443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:01.357875 4439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:01.357946 4440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:01.358556 4434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:01.359185 4434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:01.360371 4434 hybrid_clock.cc:648] HybridClock initialized: now 1786213741360355 us; error 34 us; skew 500 ppm
Aug 08 18:29:01 dist-test-slave-gp6n krb5kdc[4317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213741, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:01.363572 4434 init.cc:377] Logged in from keytab as oryx/127.29.78.129@KRBTEST.COM (short username oryx)
I20260808 18:29:01.364418 4434 webserver.cc:460] Webserver started at http://127.29.78.129:39361/ using document root <none> and password file <none>
I20260808 18:29:01.364969 4434 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:01.365060 4434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:01.365290 4434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:01.367347 4434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "01781749e0d0467da710bf3c66b32f88"
format_stamp: "Formatted at 2026-08-08 18:29:01 on dist-test-slave-gp6n"
server_key: "ad1b0d69a566dcedbdc27a2ca009ba1d"
server_key_iv: "fc60abd8e2a8ce2a5528087a9e4f4779"
server_key_version: "encryptionkey@0"
I20260808 18:29:01.367918 4434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "01781749e0d0467da710bf3c66b32f88"
format_stamp: "Formatted at 2026-08-08 18:29:01 on dist-test-slave-gp6n"
server_key: "ad1b0d69a566dcedbdc27a2ca009ba1d"
server_key_iv: "fc60abd8e2a8ce2a5528087a9e4f4779"
server_key_version: "encryptionkey@0"
I20260808 18:29:01.371619 4434 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260808 18:29:01.374060 4452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:01.375294 4434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:29:01.375449 4434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "01781749e0d0467da710bf3c66b32f88"
format_stamp: "Formatted at 2026-08-08 18:29:01 on dist-test-slave-gp6n"
server_key: "ad1b0d69a566dcedbdc27a2ca009ba1d"
server_key_iv: "fc60abd8e2a8ce2a5528087a9e4f4779"
server_key_version: "encryptionkey@0"
I20260808 18:29:01.375559 4434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:01.387840 4434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:01.391031 4434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:01.391234 4434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:01.391824 4434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:29:01.392654 4434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:01.392726 4434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:01.392799 4434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:01.392853 4434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:01.401914 4434 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:46615
I20260808 18:29:01.401942 4577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:46615 every 8 connection(s)
I20260808 18:29:01.402848 4434 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:29:01.409369 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 4434
I20260808 18:29:01.409515 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:29:01.409806 30010 external_mini_cluster.cc:1428] Setting key 873127438f4cf6c797e850068a239037
Aug 08 18:29:01 dist-test-slave-gp6n krb5kdc[4317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213741, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260808 18:29:01.417030 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:01.404672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:54653 (local address 127.29.78.190:34937)
0808 18:29:01.404821 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:01.404824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:01.405591 (+ 767us) server_negotiation.cc:407] Connection header received
0808 18:29:01.407025 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:01.407028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:01.407079 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:01.407159 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:01.408314 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:01.408878 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:01.410360 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:01.410674 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:01.410761 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:01.413462 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:01.413471 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:01.413473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:01.413496 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:01.414937 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:01.415484 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:01.415488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:01.415490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:01.415583 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:01.415930 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:01.415934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:01.415935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:01.416123 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:01.416237 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:01.416726 (+ 489us) server_negotiation.cc:299] Negotiation successful
0808 18:29:01.416856 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 18:29:01.417843 4580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:01.404991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.29.78.129:54653)
0808 18:29:01.405427 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:01.405471 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:01.406822 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:01.407325 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:01.407331 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:01.407685 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:01.408179 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:01.408192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:01.409016 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:01.409020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:01.410234 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:01.410244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:01.410828 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:01.410832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:01.411028 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:01.411782 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:01.411798 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:01.413291 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:01.415069 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:01.415076 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:01.415090 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:01.415373 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:01.415680 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:01.415684 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:01.415686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:01.415829 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:01.416236 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:01.416243 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:01.416491 (+ 248us) client_negotiation.cc:769] Sending connection context
0808 18:29:01.416682 (+ 191us) client_negotiation.cc:240] Negotiation successful
0808 18:29:01.417001 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":101,"threads_started":1}
I20260808 18:29:01.419144 4578 heartbeater.cc:347] Connected to a master server at 127.29.78.190:34937
I20260808 18:29:01.419390 4578 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:01.419878 4578 heartbeater.cc:511] Master 127.29.78.190:34937 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260808 18:29:01.421406 4366 ts_manager.cc:194] Registered new tserver with Master: 01781749e0d0467da710bf3c66b32f88 (127.29.78.129:46615)
I20260808 18:29:01.422704 4366 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:54653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.130.keytab.
Entry for principal oryx/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.130.keytab.
I20260808 18:29:01.440929 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34937
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.130.keytab
--principal=oryx/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:29:01.552166 4583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:01.552419 4583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:01.552461 4583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:01.556529 4583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:01.556586 4583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:01.556689 4583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:29:01.561775 4583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.130.keytab
--principal=oryx/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:01.562808 4583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:01.563741 4583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:29:01.573014 4583 server_base.cc:1034] running on GCE node
W20260808 18:29:01.573210 4590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:01.573187 4591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:01.573477 4593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:01.573859 4583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:01.574524 4583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:01.575686 4583 hybrid_clock.cc:648] HybridClock initialized: now 1786213741575673 us; error 24 us; skew 500 ppm
Aug 08 18:29:01 dist-test-slave-gp6n krb5kdc[4317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213741, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:01.583001 4583 init.cc:377] Logged in from keytab as oryx/127.29.78.130@KRBTEST.COM (short username oryx)
I20260808 18:29:01.583920 4583 webserver.cc:460] Webserver started at http://127.29.78.130:33869/ using document root <none> and password file <none>
I20260808 18:29:01.584471 4583 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:01.584532 4583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:01.584769 4583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:01.586905 4583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "51548778d14f48d49953bdb29879b2ca"
format_stamp: "Formatted at 2026-08-08 18:29:01 on dist-test-slave-gp6n"
server_key: "6eac9c78bffb21351ca6ecdddbfc8bcd"
server_key_iv: "4527aea866c3a722da5a5fef120d2966"
server_key_version: "encryptionkey@0"
I20260808 18:29:01.587484 4583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "51548778d14f48d49953bdb29879b2ca"
format_stamp: "Formatted at 2026-08-08 18:29:01 on dist-test-slave-gp6n"
server_key: "6eac9c78bffb21351ca6ecdddbfc8bcd"
server_key_iv: "4527aea866c3a722da5a5fef120d2966"
server_key_version: "encryptionkey@0"
I20260808 18:29:01.591253 4583 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:29:01.593741 4600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:01.595005 4583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:29:01.595141 4583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "51548778d14f48d49953bdb29879b2ca"
format_stamp: "Formatted at 2026-08-08 18:29:01 on dist-test-slave-gp6n"
server_key: "6eac9c78bffb21351ca6ecdddbfc8bcd"
server_key_iv: "4527aea866c3a722da5a5fef120d2966"
server_key_version: "encryptionkey@0"
I20260808 18:29:01.595255 4583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:01.613016 4583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:01.615926 4583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:01.616130 4583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:01.616719 4583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:29:01.617533 4583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:01.617605 4583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:01.617679 4583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:01.617722 4583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:01.626332 4583 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:38341
I20260808 18:29:01.626359 4721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:38341 every 8 connection(s)
I20260808 18:29:01.627260 4583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:29:01.627956 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 4583
I20260808 18:29:01.628095 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:29:01.628419 30010 external_mini_cluster.cc:1428] Setting key 4486b65295d10b1f368cc6f7f1d6a1e7
Aug 08 18:29:01 dist-test-slave-gp6n krb5kdc[4317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213741, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.29.78.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.131@KRBTEST.COM" created.
I20260808 18:29:01.643436 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:01.629298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:34561 (local address 127.29.78.190:34937)
0808 18:29:01.629437 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:01.629442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:01.630196 (+ 754us) server_negotiation.cc:407] Connection header received
0808 18:29:01.631646 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:01.631649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:01.631702 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:01.631780 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:01.633667 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:01.634369 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:01.636307 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:01.636529 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:01.636613 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:01.639492 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:01.639495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:01.639497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:01.639522 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:01.641140 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:01.641653 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:01.641658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:01.641659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:01.641830 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:01.642210 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:01.642213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:01.642215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:01.642521 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:01.642638 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:01.643129 (+ 491us) server_negotiation.cc:299] Negotiation successful
0808 18:29:01.643254 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 18:29:01.644407 4725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:01.629525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.29.78.130:34561)
0808 18:29:01.630057 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:01.630101 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:01.631445 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:01.632117 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:01.632129 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:01.632735 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:01.633466 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:01.633489 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:01.634582 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:01.634589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:01.636178 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:01.636191 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:01.636694 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:01.636700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:01.636946 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:01.637595 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:01.637621 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:01.639322 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:01.641262 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:01.641268 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:01.641283 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:01.641548 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:01.641960 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:01.641963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:01.641965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:01.642109 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:01.642644 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:01.642653 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:01.642894 (+ 241us) client_negotiation.cc:769] Sending connection context
0808 18:29:01.643083 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 18:29:01.643391 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":145,"threads_started":1}
I20260808 18:29:01.645720 4722 heartbeater.cc:347] Connected to a master server at 127.29.78.190:34937
I20260808 18:29:01.645952 4722 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:01.646492 4722 heartbeater.cc:511] Master 127.29.78.190:34937 requested a full tablet report, sending...
I20260808 18:29:01.647524 4366 ts_manager.cc:194] Registered new tserver with Master: 51548778d14f48d49953bdb29879b2ca (127.29.78.130:38341)
I20260808 18:29:01.648077 4366 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:34561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.131.keytab.
Entry for principal oryx/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.131.keytab.
I20260808 18:29:01.659185 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34937
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.131.keytab
--principal=oryx/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:29:01.767947 4728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:01.768217 4728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:01.768260 4728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:01.772197 4728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:01.772250 4728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:01.772306 4728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:29:01.777693 4728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.131.keytab
--principal=oryx/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:01.778832 4728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:01.779899 4728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:01.789503 4736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:01.789467 4733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:01.789618 4728 server_base.cc:1034] running on GCE node
W20260808 18:29:01.789799 4738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:01.790189 4728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:01.790874 4728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:01.792054 4728 hybrid_clock.cc:648] HybridClock initialized: now 1786213741792039 us; error 29 us; skew 500 ppm
Aug 08 18:29:01 dist-test-slave-gp6n krb5kdc[4317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213741, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:01.794842 4728 init.cc:377] Logged in from keytab as oryx/127.29.78.131@KRBTEST.COM (short username oryx)
I20260808 18:29:01.795754 4728 webserver.cc:460] Webserver started at http://127.29.78.131:45987/ using document root <none> and password file <none>
I20260808 18:29:01.796317 4728 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:01.796406 4728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:01.796645 4728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:01.798722 4728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "5d47d6d7c4d445dbaebedcaae667f1de"
format_stamp: "Formatted at 2026-08-08 18:29:01 on dist-test-slave-gp6n"
server_key: "f38c32d14a1dffb89710779662107fc9"
server_key_iv: "87d36d20de401b10d61752381eca8901"
server_key_version: "encryptionkey@0"
I20260808 18:29:01.799372 4728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "5d47d6d7c4d445dbaebedcaae667f1de"
format_stamp: "Formatted at 2026-08-08 18:29:01 on dist-test-slave-gp6n"
server_key: "f38c32d14a1dffb89710779662107fc9"
server_key_iv: "87d36d20de401b10d61752381eca8901"
server_key_version: "encryptionkey@0"
I20260808 18:29:01.802981 4728 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 18:29:01.805351 4746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:01.806612 4728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:29:01.806734 4728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "5d47d6d7c4d445dbaebedcaae667f1de"
format_stamp: "Formatted at 2026-08-08 18:29:01 on dist-test-slave-gp6n"
server_key: "f38c32d14a1dffb89710779662107fc9"
server_key_iv: "87d36d20de401b10d61752381eca8901"
server_key_version: "encryptionkey@0"
I20260808 18:29:01.806838 4728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:01.819571 4728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:01.822347 4728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:01.822588 4728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:01.823166 4728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:29:01.824000 4728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:01.824069 4728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:01.824136 4728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:01.824183 4728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:01.833252 4728 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:41395
I20260808 18:29:01.833308 4878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:41395 every 8 connection(s)
I20260808 18:29:01.834153 4728 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:29:01.835115 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 4728
I20260808 18:29:01.835251 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:29:01.835544 30010 external_mini_cluster.cc:1428] Setting key d9a618fb6037d592bd3a5dbc483a55e3
Aug 08 18:29:01 dist-test-slave-gp6n krb5kdc[4317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213741, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260808 18:29:01.848644 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:01.836088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:42399 (local address 127.29.78.190:34937)
0808 18:29:01.836225 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:01.836232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:01.837153 (+ 921us) server_negotiation.cc:407] Connection header received
0808 18:29:01.838763 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:01.838765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:01.838813 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:01.838884 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:01.840022 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:01.840658 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:01.841788 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:01.842013 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:01.842103 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:01.844879 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:01.844883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:01.844885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:01.844913 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:01.846639 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:01.847097 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:01.847101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:01.847102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:01.847183 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:01.847480 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:01.847482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:01.847483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:01.847682 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:01.847767 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:01.848304 (+ 537us) server_negotiation.cc:299] Negotiation successful
0808 18:29:01.848463 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 18:29:01.849680 4881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:01.836519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.29.78.131:42399)
0808 18:29:01.837004 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:01.837062 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:01.838574 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:01.839012 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:01.839019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:01.839369 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:01.839859 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:01.839873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:01.840774 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:01.840777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:01.841662 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:01.841671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:01.842147 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:01.842150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:01.842285 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:01.842888 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:01.842904 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:01.844701 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:01.846738 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:01.846743 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:01.846755 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:01.847006 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:01.847263 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:01.847265 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:01.847266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:01.847397 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:01.847771 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:01.847776 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:01.848000 (+ 224us) client_negotiation.cc:769] Sending connection context
0808 18:29:01.848220 (+ 220us) client_negotiation.cc:240] Negotiation successful
0808 18:29:01.848571 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":105,"threads_started":1}
I20260808 18:29:01.850840 4879 heartbeater.cc:347] Connected to a master server at 127.29.78.190:34937
I20260808 18:29:01.851065 4879 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:01.851617 4879 heartbeater.cc:511] Master 127.29.78.190:34937 requested a full tablet report, sending...
I20260808 18:29:01.852581 4366 ts_manager.cc:194] Registered new tserver with Master: 5d47d6d7c4d445dbaebedcaae667f1de (127.29.78.131:41395)
I20260808 18:29:01.853142 4366 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:42399
I20260808 18:29:01.861347 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.29.78.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.189.keytab.
Entry for principal oryx/127.29.78.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.189.keytab.
I20260808 18:29:01.894142 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.189:32899
--webserver_interface=127.29.78.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.189.keytab
--principal=oryx/127.29.78.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:34937,127.29.78.189:32899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:29:02.005030 4884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:02.005317 4884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:02.005358 4884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:02.009461 4884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:29:02.009581 4884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:02.009620 4884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:02.009686 4884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:29:02.009725 4884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:29:02.015245 4884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:34937,127.29.78.189:32899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.189.keytab
--principal=oryx/127.29.78.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.189:32899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.78.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
W20260808 18:29:02.015528 4884 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.29.78.190:34937,127.29.78.189:32899'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260808 18:29:02.016482 4884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:02.017562 4884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:02.026190 4890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:02.026191 4891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:02.026279 4893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:02.026459 4884 server_base.cc:1034] running on GCE node
I20260808 18:29:02.027091 4884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:02.028187 4884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:02.029395 4884 hybrid_clock.cc:648] HybridClock initialized: now 1786213742029376 us; error 31 us; skew 500 ppm
Aug 08 18:29:02 dist-test-slave-gp6n krb5kdc[4317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213742, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:02.032608 4884 init.cc:377] Logged in from keytab as oryx/127.29.78.189@KRBTEST.COM (short username oryx)
I20260808 18:29:02.033536 4884 webserver.cc:460] Webserver started at http://127.29.78.189:33505/ using document root <none> and password file <none>
I20260808 18:29:02.034097 4884 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:02.034188 4884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:02.034489 4884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:02.036582 4884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/data/instance:
uuid: "a692e2279f124613a8b62dc5c5cda6ce"
format_stamp: "Formatted at 2026-08-08 18:29:02 on dist-test-slave-gp6n"
server_key: "a111bcfe1fb8ab610050f8a263ad3e98"
server_key_iv: "c8a1029b489df5ef26f3a7e3a7816221"
server_key_version: "encryptionkey@0"
I20260808 18:29:02.037177 4884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/wal/instance:
uuid: "a692e2279f124613a8b62dc5c5cda6ce"
format_stamp: "Formatted at 2026-08-08 18:29:02 on dist-test-slave-gp6n"
server_key: "a111bcfe1fb8ab610050f8a263ad3e98"
server_key_iv: "c8a1029b489df5ef26f3a7e3a7816221"
server_key_version: "encryptionkey@0"
I20260808 18:29:02.041090 4884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260808 18:29:02.043632 4901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:02.044978 4884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:29:02.045112 4884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/wal
uuid: "a692e2279f124613a8b62dc5c5cda6ce"
format_stamp: "Formatted at 2026-08-08 18:29:02 on dist-test-slave-gp6n"
server_key: "a111bcfe1fb8ab610050f8a263ad3e98"
server_key_iv: "c8a1029b489df5ef26f3a7e3a7816221"
server_key_version: "encryptionkey@0"
I20260808 18:29:02.045228 4884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:02.073547 4884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:02.077076 4884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:02.077271 4884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:02.084908 4884 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.189:32899
I20260808 18:29:02.085237 4953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.189:32899 every 8 connection(s)
I20260808 18:29:02.085933 4884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/data/info.pb
I20260808 18:29:02.088713 4954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:02.091609 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 4884
I20260808 18:29:02.093302 4954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 18:29:02 dist-test-slave-gp6n krb5kdc[4317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213742, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.189@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
Aug 08 18:29:02 dist-test-slave-gp6n krb5kdc[4317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.78.189@KRBTEST.COM
I20260808 18:29:02.105922 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.095127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48852 (local address 127.29.78.190:34937)
0808 18:29:02.095258 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.095263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.095621 (+ 358us) server_negotiation.cc:407] Connection header received
0808 18:29:02.095823 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.095826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.095879 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.095973 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:02.096725 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.097243 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.098289 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.098601 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.098716 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.101252 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:02.101259 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:02.101261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:02.101290 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:02.103434 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.104183 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.104186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.104188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.104288 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.104710 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.104715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.104717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.104976 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:02.105102 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.105588 (+ 486us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.105720 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 18:29:02.106792 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.093298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53972 (local address 127.29.78.189:32899)
0808 18:29:02.093746 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.093771 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.093809 (+ 38us) server_negotiation.cc:407] Connection header received
0808 18:29:02.094521 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.094541 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.094866 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.095202 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:02.096104 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.097118 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.098021 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.098390 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.098618 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.101815 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:02.101821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:02.101831 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:02.101851 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:02.104158 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.104608 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.104614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.104616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.104698 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.104988 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.104993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.104994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.105326 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:02.105532 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.105769 (+ 237us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.105932 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":286,"thread_start_us":136,"threads_started":1}
I20260808 18:29:02.109901 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.095062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.0.0.1:48852)
0808 18:29:02.095495 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.095545 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:02.095711 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:02.096025 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:02.096031 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:02.096384 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:02.096581 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.096591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.097382 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.097385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.098151 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.098161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.098754 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.098758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.098964 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.099493 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:02.099509 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:02.101096 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:02.103606 (+ 2510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.103612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.103623 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.104056 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.104423 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.104426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.104428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.104598 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.105099 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:02.105106 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:02.105348 (+ 242us) client_negotiation.cc:769] Sending connection context
0808 18:29:02.105537 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 18:29:02.105864 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":29,"thread_start_us":126,"threads_started":1}
I20260808 18:29:02.111291 4954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 18:29:02 dist-test-slave-gp6n krb5kdc[4317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213742, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.189@KRBTEST.COM for oryx/127.29.78.189@KRBTEST.COM
I20260808 18:29:02.120258 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.111747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:32899 (local address 127.0.0.1:53980)
0808 18:29:02.111873 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.111889 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:02.111967 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:02.112177 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:02.112179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:02.112362 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:02.112488 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.112496 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.113530 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.113534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.114499 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.114510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.114973 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.114977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.115085 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.115572 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:02.115587 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:02.117002 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:02.118859 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.118863 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.118866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.119054 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.119326 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.119329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.119331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.119416 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.119761 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:02.119763 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:02.119838 (+ 75us) client_negotiation.cc:769] Sending connection context
0808 18:29:02.119954 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 18:29:02.120084 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 18:29:02.120265 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.111853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53980 (local address 127.29.78.189:32899)
0808 18:29:02.111951 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.111954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.111966 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:29:02.112051 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.112053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.112083 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.112143 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:02.112602 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.113424 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.114622 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.114863 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.114950 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.117143 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:02.117146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:02.117147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:02.117165 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:02.118738 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.119157 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.119160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.119162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.119233 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.119508 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.119511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.119513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.119671 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:02.119771 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.119997 (+ 226us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.120139 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":37}
I20260808 18:29:02.123152 4954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce: Bootstrap starting.
I20260808 18:29:02.125435 4954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:02.126145 4954 log.cc:824] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce: Log is configured to *not* fsync() on all Append() calls
I20260808 18:29:02.128082 4954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce: No bootstrap required, opened a new log
I20260808 18:29:02.131256 4954 raft_consensus.cc:348] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } }
I20260808 18:29:02.131520 4954 raft_consensus.cc:374] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:02.131568 4954 raft_consensus.cc:729] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a692e2279f124613a8b62dc5c5cda6ce, State: Initialized, Role: FOLLOWER
I20260808 18:29:02.132050 4954 consensus_queue.cc:260] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } }
I20260808 18:29:02.132566 4961 sys_catalog.cc:455] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } } }
I20260808 18:29:02.132678 4961 sys_catalog.cc:458] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:29:02.133466 4954 sys_catalog.cc:564] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: configured and running, proceeding with master startup.
W20260808 18:29:02.138963 4975 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:29:02.139058 4975 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:29:02.141237 4884 master_runner.cc:428] Detected that this master a692e2279f124613a8b62dc5c5cda6ce is joining an existing cluster
I20260808 18:29:02.141304 4884 master_runner.cc:432] Initiating AddMaster RPC to add 127.29.78.189:32899
I20260808 18:29:02.152323 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.143647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53982 (local address 127.29.78.189:32899)
0808 18:29:02.143773 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.143777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.144460 (+ 683us) server_negotiation.cc:407] Connection header received
0808 18:29:02.144577 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.144579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.144618 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.144696 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:02.145244 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.145730 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.146865 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.147218 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.147329 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.148857 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:02.148861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:02.148863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:02.148896 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:02.150652 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.151109 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.151113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.151115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.151215 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.151487 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.151490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.151491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.151693 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:02.151757 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.152048 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.152158 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":47,"spinlock_wait_cycles":10880}
I20260808 18:29:02.152633 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.143565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.0.0.1:48866)
0808 18:29:02.143965 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.143982 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:02.144099 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:02.144383 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:02.144387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:02.144643 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:02.145022 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.145049 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.146128 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.146131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.146942 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.146949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.147546 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.147551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.147660 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.148224 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:02.148237 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:02.148689 (+ 452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:02.150978 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.150981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.150983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.151204 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.151482 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.151487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.151489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.151605 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.151979 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:02.151982 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:02.152084 (+ 102us) client_negotiation.cc:769] Sending connection context
0808 18:29:02.152191 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 18:29:02.152326 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":200,"spinlock_wait_cycles":8576,"thread_start_us":90,"threads_started":1}
I20260808 18:29:02.152793 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.143772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48866 (local address 127.29.78.190:34937)
0808 18:29:02.143910 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.143915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.144069 (+ 154us) server_negotiation.cc:407] Connection header received
0808 18:29:02.144207 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.144211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.144261 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.144351 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:02.145175 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.146013 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.147057 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.147312 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.147414 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.148958 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:02.148961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:02.148964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:02.148987 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:02.150861 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.151298 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.151302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.151304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.151391 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.151698 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.151703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.151704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.151883 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:02.151990 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.152245 (+ 255us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.152360 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 18:29:02.152770 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.144028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:32899 (local address 127.0.0.1:53982)
0808 18:29:02.144382 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.144395 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:02.144484 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:02.144736 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:02.144741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:02.144990 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:02.145100 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.145109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.145844 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.145847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.146740 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.146750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.147358 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.147364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.147460 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.148041 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:02.148052 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:02.148689 (+ 637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:02.150770 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.150773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.150775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.151004 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.151317 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.151320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.151322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.151401 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.151812 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:02.151814 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:02.151882 (+ 68us) client_negotiation.cc:769] Sending connection context
0808 18:29:02.152453 (+ 571us) client_negotiation.cc:240] Negotiation successful
0808 18:29:02.152570 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":2,"spinlock_wait_cycles":2432,"thread_start_us":106,"threads_started":1}
Aug 08 18:29:02 dist-test-slave-gp6n krb5kdc[4317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213741, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.190@KRBTEST.COM for oryx/127.29.78.189@KRBTEST.COM
I20260808 18:29:02.167898 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.157230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53996 (local address 127.29.78.189:32899)
0808 18:29:02.157340 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.157344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.157684 (+ 340us) server_negotiation.cc:407] Connection header received
0808 18:29:02.157857 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.157860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.157917 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.157996 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:02.158628 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.159365 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.160621 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.160986 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.161145 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.163811 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:02.163815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:02.163819 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:02.163844 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:02.165444 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.166063 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.166066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.166069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.166163 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.166559 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.166564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.166569 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.167089 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:02.167168 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.167627 (+ 459us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.167737 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 18:29:02.168399 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.157191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:32899 (local address 127.0.0.1:53996)
0808 18:29:02.157555 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.157606 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:02.157732 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:02.158041 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:02.158045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:02.158359 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:02.158521 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.158527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.159498 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.159501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.160490 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.160501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.161276 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.161283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.161433 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.161998 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:02.162019 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:02.163667 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:02.165566 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.165580 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.165589 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.165936 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.166281 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.166286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.166289 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.166431 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.167256 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:02.167263 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:02.167392 (+ 129us) client_negotiation.cc:769] Sending connection context
0808 18:29:02.168031 (+ 639us) client_negotiation.cc:240] Negotiation successful
0808 18:29:02.168234 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":122,"threads_started":1}
I20260808 18:29:02.170003 4366 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.29.78.189:32899: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce"
member_type: NON_VOTER
last_known_addr {
host: "127.29.78.189"
port: 32899
}
attrs {
promote: true
}
}
dest_uuid: "181bd4194326418da33169a4f27346f8"
cas_config_opid_index: -1
WARNING: no policy specified for oryx/127.29.78.188@KRBTEST.COM; defaulting to no policy
I20260808 18:29:02.170778 4366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: true } }
I20260808 18:29:02.172601 4990 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a692e2279f124613a8b62dc5c5cda6ce (127.29.78.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: true } } }
I20260808 18:29:02.173221 4928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:02.174257 4990 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.29.78.189:32899
I20260808 18:29:02.174475 4928 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 1 FOLLOWER]: Refusing update from remote peer 181bd4194326418da33169a4f27346f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 18:29:02.174484 4990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "181bd4194326418da33169a4f27346f8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: true } } }
I20260808 18:29:02.174574 4990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.188@KRBTEST.COM" created.
I20260808 18:29:02.175148 4990 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:29:02.177265 4884 master.cc:537] Master@127.29.78.189:32899 shutting down...
I20260808 18:29:02.181151 4928 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a692e2279f124613a8b62dc5c5cda6ce (127.29.78.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: true } } }
I20260808 18:29:02.183213 4961 sys_catalog.cc:455] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 181bd4194326418da33169a4f27346f8. Latest consensus state: current_term: 1 leader_uuid: "181bd4194326418da33169a4f27346f8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: true } } }
I20260808 18:29:02.183346 4961 sys_catalog.cc:458] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: This master's current role is: LEARNER
I20260808 18:29:02.183319 4996 sys_catalog.cc:455] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "181bd4194326418da33169a4f27346f8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: true } } }
I20260808 18:29:02.183564 4996 sys_catalog.cc:458] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: This master's current role is: LEARNER
I20260808 18:29:02.183663 4997 sys_catalog.cc:455] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "181bd4194326418da33169a4f27346f8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: true } } }
I20260808 18:29:02.183756 4997 sys_catalog.cc:458] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: This master's current role is: LEARNER
I20260808 18:29:02.188514 4994 mvcc.cc:205] Tried to move back new op lower bound from 7316331483813842944 to 7316331483670884352. Current Snapshot: MvccSnapshot[applied={T|T < 7316331483709820928}]
I20260808 18:29:02.190274 4884 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 1 LEARNER]: Raft consensus shutting down.
I20260808 18:29:02.190611 4884 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 1 LEARNER]: Raft consensus is shut down!
I20260808 18:29:02.190721 4884 tablet_replica.cc:330] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.188.keytab.
Entry for principal oryx/127.29.78.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.188.keytab.
I20260808 18:29:02.197206 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.188:43147
--webserver_interface=127.29.78.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.188.keytab
--principal=oryx/127.29.78.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:34937,127.29.78.189:32899,127.29.78.188:43147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:29:02.205785 4884 master.cc:552] Master@127.29.78.189:32899 shutdown complete.
I20260808 18:29:02.205847 4884 master_runner.cc:305] Clearing existing system tablet
I20260808 18:29:02.206102 4884 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:02.206159 4884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:02.207568 4884 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 18:29:02.208523 5003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:02.208778 4884 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:02.208910 4884 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 18:29:02.208952 4884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/wal
uuid: "a692e2279f124613a8b62dc5c5cda6ce"
format_stamp: "Formatted at 2026-08-08 18:29:02 on dist-test-slave-gp6n"
server_key: "a111bcfe1fb8ab610050f8a263ad3e98"
server_key_iv: "c8a1029b489df5ef26f3a7e3a7816221"
server_key_version: "encryptionkey@0"
I20260808 18:29:02.210785 4884 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:29:02.213075 4884 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 18:29:02.213164 4884 log.cc:1196] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260808 18:29:02.213459 4884 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce: Deleting consensus metadata
I20260808 18:29:02.213626 4884 master_runner.cc:315] Copying system tablet from 127.29.78.190:34937
I20260808 18:29:02.214959 4884 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce: tablet copy: Beginning tablet copy session from remote peer at address 127.29.78.190:34937
I20260808 18:29:02.225652 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.215676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48876 (local address 127.29.78.190:34937)
0808 18:29:02.215810 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.215815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.216054 (+ 239us) server_negotiation.cc:407] Connection header received
0808 18:29:02.216251 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.216255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.216307 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.216365 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:02.217013 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.217841 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.219083 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.219426 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.219501 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.221238 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:02.221242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:02.221245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:02.221295 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:02.223547 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.224276 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.224280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.224282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.224398 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.224702 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.224706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.224708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.224937 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:02.225012 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.225356 (+ 344us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.225480 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 18:29:02.226217 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.215593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.0.0.1:48876)
0808 18:29:02.215954 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.215969 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:02.216135 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:02.216512 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:02.216516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:02.216734 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:02.216857 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.216864 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.217998 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.218001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.218942 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.218952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.219682 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.219685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.219777 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.220373 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:02.220414 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:02.221031 (+ 617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:02.223712 (+ 2681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.223715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.223718 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.224112 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.224512 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.224515 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.224517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.224602 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.225085 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:02.225088 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:02.225178 (+ 90us) client_negotiation.cc:769] Sending connection context
0808 18:29:02.225865 (+ 687us) client_negotiation.cc:240] Negotiation successful
0808 18:29:02.226040 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":90,"threads_started":1}
I20260808 18:29:02.227015 4387 tablet_copy_service.cc:140] P 181bd4194326418da33169a4f27346f8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a692e2279f124613a8b62dc5c5cda6ce ({username='oryx', principal='oryx/127.29.78.189@KRBTEST.COM'} at 127.0.0.1:48876)
I20260808 18:29:02.227197 4387 tablet_copy_service.cc:161] P 181bd4194326418da33169a4f27346f8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a692e2279f124613a8b62dc5c5cda6ce at {username='oryx', principal='oryx/127.29.78.189@KRBTEST.COM'} at 127.0.0.1:48876: session id = a692e2279f124613a8b62dc5c5cda6ce-00000000000000000000000000000000
I20260808 18:29:02.229130 4387 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 18:29:02.231096 4884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 18:29:02.233610 4884 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce: tablet copy: Starting download of 0 data blocks...
I20260808 18:29:02.233799 4884 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce: tablet copy: Starting download of 1 WAL segments...
I20260808 18:29:02.235890 4884 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 18:29:02.237617 4387 tablet_copy_service.cc:342] P 181bd4194326418da33169a4f27346f8: Request end of tablet copy session a692e2279f124613a8b62dc5c5cda6ce-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.29.78.189@KRBTEST.COM'} at 127.0.0.1:48876
I20260808 18:29:02.237737 4387 tablet_copy_service.cc:433] P 181bd4194326418da33169a4f27346f8: ending tablet copy session a692e2279f124613a8b62dc5c5cda6ce-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a692e2279f124613a8b62dc5c5cda6ce
I20260808 18:29:02.245605 4884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:02.247320 5010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:02.247385 5011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:02.251952 5013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:02.252089 4884 server_base.cc:1034] running on GCE node
I20260808 18:29:02.252348 4884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:02.252655 4884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:02.253806 4884 hybrid_clock.cc:648] HybridClock initialized: now 1786213742253782 us; error 40 us; skew 500 ppm
Aug 08 18:29:02 dist-test-slave-gp6n krb5kdc[4317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213742, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:02.255463 4884 init.cc:377] Logged in from keytab as oryx/127.29.78.189@KRBTEST.COM (short username oryx)
I20260808 18:29:02.255918 4884 webserver.cc:460] Webserver started at http://127.29.78.189:46039/ using document root <none> and password file <none>
I20260808 18:29:02.256106 4884 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:02.256150 4884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:02.257479 4884 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 18:29:02.258576 5020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:02.258842 4884 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 18:29:02.258920 4884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/wal
uuid: "a692e2279f124613a8b62dc5c5cda6ce"
format_stamp: "Formatted at 2026-08-08 18:29:02 on dist-test-slave-gp6n"
server_key: "a111bcfe1fb8ab610050f8a263ad3e98"
server_key_iv: "c8a1029b489df5ef26f3a7e3a7816221"
server_key_version: "encryptionkey@0"
I20260808 18:29:02.259003 4884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:02.277988 4884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:02.280550 4884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:02.285754 4884 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.189:32899
I20260808 18:29:02.285877 5077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.189:32899 every 8 connection(s)
I20260808 18:29:02.286355 4884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-1/data/info.pb
I20260808 18:29:02.288559 5078 sys_catalog.cc:263] Verifying existing consensus state
I20260808 18:29:02.289836 5078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce: Bootstrap starting.
I20260808 18:29:02.298583 5078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:29:02.298923 5078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce: Bootstrap complete.
I20260808 18:29:02.299561 5078 raft_consensus.cc:348] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: true } }
I20260808 18:29:02.299721 5078 raft_consensus.cc:729] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a692e2279f124613a8b62dc5c5cda6ce, State: Initialized, Role: LEARNER
I20260808 18:29:02.299860 5078 consensus_queue.cc:260] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: true } }
I20260808 18:29:02.300225 5082 sys_catalog.cc:455] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: true } } }
I20260808 18:29:02.300360 5078 sys_catalog.cc:564] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:29:02.300454 5082 sys_catalog.cc:458] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: This master's current role is: LEARNER
I20260808 18:29:02.304242 5093 catalog_manager.cc:1259] Loaded cluster ID: 79b2d9f0f7a04df6b2088bd4ca8342eb
I20260808 18:29:02.304307 5093 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce: loading cluster ID for follower catalog manager: success
I20260808 18:29:02.306536 5093 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce: acquiring CA information for follower catalog manager: success
I20260808 18:29:02.307931 5093 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 18:29:02.334159 5001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:02.334435 5001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:02.334533 5001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:02.338387 5001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:29:02.338456 5001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:02.338476 5001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:02.338492 5001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:29:02.338508 5001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:29:02.343878 5001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:34937,127.29.78.189:32899,127.29.78.188:43147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.188.keytab
--principal=oryx/127.29.78.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.188:43147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.78.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:02.345279 5001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:02.346570 5001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:02.355225 5098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:02.355216 5099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:02.355217 5102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:02.355661 5001 server_base.cc:1034] running on GCE node
I20260808 18:29:02.356199 5001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:02.357271 5001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:02.358502 5001 hybrid_clock.cc:648] HybridClock initialized: now 1786213742358484 us; error 26 us; skew 500 ppm
Aug 08 18:29:02 dist-test-slave-gp6n krb5kdc[4317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213742, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:02.361299 5001 init.cc:377] Logged in from keytab as oryx/127.29.78.188@KRBTEST.COM (short username oryx)
I20260808 18:29:02.362195 5001 webserver.cc:460] Webserver started at http://127.29.78.188:34983/ using document root <none> and password file <none>
I20260808 18:29:02.362798 5001 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:02.362861 5001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:02.363039 5001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:02.365043 5001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/data/instance:
uuid: "12f6a1d683a94e278adf262ca5911ca6"
format_stamp: "Formatted at 2026-08-08 18:29:02 on dist-test-slave-gp6n"
server_key: "5a2de8bbfa1a2cf42ebe42a26238ddfa"
server_key_iv: "af4cdf8bdbf4b2f84f9459ea4ddcd582"
server_key_version: "encryptionkey@0"
I20260808 18:29:02.365567 5001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/wal/instance:
uuid: "12f6a1d683a94e278adf262ca5911ca6"
format_stamp: "Formatted at 2026-08-08 18:29:02 on dist-test-slave-gp6n"
server_key: "5a2de8bbfa1a2cf42ebe42a26238ddfa"
server_key_iv: "af4cdf8bdbf4b2f84f9459ea4ddcd582"
server_key_version: "encryptionkey@0"
I20260808 18:29:02.369376 5001 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 18:29:02.371954 5111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:02.373232 5001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:29:02.373378 5001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/wal
uuid: "12f6a1d683a94e278adf262ca5911ca6"
format_stamp: "Formatted at 2026-08-08 18:29:02 on dist-test-slave-gp6n"
server_key: "5a2de8bbfa1a2cf42ebe42a26238ddfa"
server_key_iv: "af4cdf8bdbf4b2f84f9459ea4ddcd582"
server_key_version: "encryptionkey@0"
I20260808 18:29:02.373488 5001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:02.389048 5001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:02.392323 5001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:02.392527 5001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:02.400089 5001 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.188:43147
I20260808 18:29:02.400405 5167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.188:43147 every 8 connection(s)
I20260808 18:29:02.401280 5001 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/data/info.pb
I20260808 18:29:02.404279 5168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:02.407287 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 5001
I20260808 18:29:02.408279 5168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 18:29:02 dist-test-slave-gp6n krb5kdc[4317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213742, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.188@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
Aug 08 18:29:02 dist-test-slave-gp6n krb5kdc[4317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.78.188@KRBTEST.COM
I20260808 18:29:02.421783 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.410207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48880 (local address 127.29.78.190:34937)
0808 18:29:02.410337 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.410341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.410606 (+ 265us) server_negotiation.cc:407] Connection header received
0808 18:29:02.410905 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.410908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.410962 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.411051 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:02.412112 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.412913 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.414332 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.414724 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.414836 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.417930 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:02.417934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:02.417936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:02.417960 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:02.419607 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.420245 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.420247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.420248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.420319 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.420694 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.420699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.420701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.420887 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:02.420993 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.421478 (+ 485us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.421597 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 18:29:02.422750 5171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.410007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.0.0.1:48880)
0808 18:29:02.410437 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.410490 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:02.410705 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:02.411133 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:02.411138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:02.411610 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:02.411975 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.411987 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.413060 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.413063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.414218 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.414229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.414906 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.414909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.415108 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.415929 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:02.415948 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:02.417763 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:02.419719 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.419728 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.419737 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.420151 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.420451 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.420456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.420458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.420600 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.420990 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:02.420997 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:02.421244 (+ 247us) client_negotiation.cc:769] Sending connection context
0808 18:29:02.421423 (+ 179us) client_negotiation.cc:240] Negotiation successful
0808 18:29:02.421736 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":29,"thread_start_us":99,"threads_started":1}
I20260808 18:29:02.422748 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.408819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50244 (local address 127.29.78.188:43147)
0808 18:29:02.409241 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.409260 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.409296 (+ 36us) server_negotiation.cc:407] Connection header received
0808 18:29:02.410258 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.410281 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.410709 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.410949 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:02.411926 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.413198 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.414330 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.414942 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.415130 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.418351 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:02.418357 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:02.418367 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:02.418387 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:02.420227 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.420675 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.420683 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.420684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.420763 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.421051 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.421055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.421056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.421304 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:02.421496 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.421752 (+ 256us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.421896 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":282,"thread_start_us":149,"threads_started":1}
I20260808 18:29:02.424685 5168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:29:02.425551 4578 heartbeater.cc:503] Master 127.29.78.190:34937 was elected leader, sending a full tablet report...
Aug 08 18:29:02 dist-test-slave-gp6n krb5kdc[4317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213742, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.188@KRBTEST.COM for oryx/127.29.78.189@KRBTEST.COM
I20260808 18:29:02.433466 5171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.425149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:32899 (local address 127.0.0.1:53998)
0808 18:29:02.425256 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.425268 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:02.425327 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:02.425816 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:02.425819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:02.425989 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:02.426103 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.426111 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.427141 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.427143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.427680 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.427688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.428140 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.428143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.428233 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.428690 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:02.428702 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:02.429944 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:02.431985 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.431988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.431990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.432213 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.432464 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.432466 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.432468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.432557 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.432954 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:02.432957 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:02.433052 (+ 95us) client_negotiation.cc:769] Sending connection context
0808 18:29:02.433160 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 18:29:02.433298 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 18:29:02.433507 5172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.425277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53998 (local address 127.29.78.189:32899)
0808 18:29:02.425588 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.425592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.425603 (+ 11us) server_negotiation.cc:407] Connection header received
0808 18:29:02.425640 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.425642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.425702 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.425802 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:02.426208 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.427048 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.427778 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.428006 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.428098 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.430112 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:02.430116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:02.430118 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:02.430144 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:02.431885 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.432303 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.432305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.432306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.432383 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.432644 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.432646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.432647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.432814 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:02.432925 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.433227 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.433346 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":158,"threads_started":1}
I20260808 18:29:02.434326 5168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 18:29:02 dist-test-slave-gp6n krb5kdc[4317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213742, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.188@KRBTEST.COM for oryx/127.29.78.188@KRBTEST.COM
I20260808 18:29:02.443519 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.434891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50252 (local address 127.29.78.188:43147)
0808 18:29:02.434982 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.434989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.435037 (+ 48us) server_negotiation.cc:407] Connection header received
0808 18:29:02.435182 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.435185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.435229 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.435323 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:02.435700 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.436246 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.437073 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.438012 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.438143 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.440186 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:02.440190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:02.440193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:02.440219 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:02.441561 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.441995 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.442002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.442004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.442082 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.442377 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.442381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.442382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.442791 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:02.442939 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.443210 (+ 271us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.443376 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":39}
I20260808 18:29:02.443519 5171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.434813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43147 (local address 127.0.0.1:50252)
0808 18:29:02.434973 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.434989 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:02.435079 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:02.435326 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:02.435329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:02.435513 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:02.435579 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.435584 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.436384 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.436391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.436945 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.436953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.438139 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.438143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.438287 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.438750 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:02.438762 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:02.440052 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:02.441675 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.441682 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.441686 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.441894 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.442182 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.442185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.442187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.442269 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.442925 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:02.442928 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:02.443037 (+ 109us) client_negotiation.cc:769] Sending connection context
0808 18:29:02.443150 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 18:29:02.443308 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":39}
I20260808 18:29:02.447060 5168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6: Bootstrap starting.
I20260808 18:29:02.449467 5168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:02.450237 5168 log.cc:824] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6: Log is configured to *not* fsync() on all Append() calls
I20260808 18:29:02.452243 5168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6: No bootstrap required, opened a new log
I20260808 18:29:02.455480 5168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } }
I20260808 18:29:02.455701 5168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:02.455746 5168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f6a1d683a94e278adf262ca5911ca6, State: Initialized, Role: FOLLOWER
I20260808 18:29:02.456156 5168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } }
I20260808 18:29:02.456574 5174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } } }
I20260808 18:29:02.456701 5174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:29:02.457322 5168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 18:29:02.462509 5185 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:29:02.462601 5185 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:29:02.465910 5001 master_runner.cc:428] Detected that this master 12f6a1d683a94e278adf262ca5911ca6 is joining an existing cluster
I20260808 18:29:02.465993 5001 master_runner.cc:432] Initiating AddMaster RPC to add 127.29.78.188:43147
I20260808 18:29:02.476012 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 4434
I20260808 18:29:02.482700 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.468652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50268 (local address 127.29.78.188:43147)
0808 18:29:02.468748 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.468752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.469521 (+ 769us) server_negotiation.cc:407] Connection header received
0808 18:29:02.469667 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.469670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.469707 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.469751 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:02.470294 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.471064 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.472284 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.472600 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.472670 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.474116 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:02.474118 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:02.474120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:02.474139 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:02.475726 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.481436 (+ 5710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.481441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.481443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.481528 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.481810 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.481813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.481815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.481998 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:02.482100 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.482360 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.482525 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":39}
I20260808 18:29:02.482735 5203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.468824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43147 (local address 127.0.0.1:50268)
0808 18:29:02.469449 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.469462 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:02.469580 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:02.469865 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:02.469868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:02.470069 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:02.470179 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.470186 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.471177 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.471181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.472161 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.472172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.472817 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.472820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.472907 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.473471 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:02.473485 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:02.473984 (+ 499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:02.475936 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.475939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.475941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.481301 (+ 5360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.481623 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.481628 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.481630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.481717 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.482098 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:02.482101 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:02.482191 (+ 90us) client_negotiation.cc:769] Sending connection context
0808 18:29:02.482290 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 18:29:02.482586 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":147,"thread_start_us":80,"threads_started":1}
I20260808 18:29:02.484707 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:46615
--local_ip_for_outbound_sockets=127.29.78.129
--tserver_master_addrs=127.29.78.190:34937
--webserver_port=39361
--webserver_interface=127.29.78.129
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.129.keytab
--principal=oryx/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.78.190:34937,127.29.78.189:32899
--tserver_master_addrs=127.29.78.190:34937,127.29.78.189:32899,127.29.78.188:43147 with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:29:02.508179 5172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.468313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54014 (local address 127.29.78.189:32899)
0808 18:29:02.468441 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.468446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.469198 (+ 752us) server_negotiation.cc:407] Connection header received
0808 18:29:02.469295 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.469298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.469371 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.469452 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:02.470015 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.470800 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.472470 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.472813 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.472877 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.503880 (+ 31003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:02.503886 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:02.503889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:02.503920 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:02.506078 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.506679 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.506683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.506684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.506769 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.507112 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.507116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.507118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.507373 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:02.507450 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.507822 (+ 372us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.507983 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 18:29:02.508420 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.468579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48896 (local address 127.29.78.190:34937)
0808 18:29:02.468707 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.468712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.468726 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:29:02.468766 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.468769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.468818 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.468908 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:02.469468 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.470281 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.472414 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.472748 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.472821 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.504609 (+ 31788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:02.504615 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:02.504620 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:02.504661 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:02.506523 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.507015 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.507020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.507022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.507126 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.507460 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.507466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.507471 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.507686 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:02.507761 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.508121 (+ 360us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.508244 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 18:29:02.508828 5193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.468223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:32899 (local address 127.0.0.1:54014)
0808 18:29:02.469100 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.469114 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:02.469213 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:02.469560 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:02.469563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:02.469778 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:02.469887 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.469894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.470926 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.470929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.472351 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.472362 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.473003 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.473005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.473148 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.503153 (+ 30005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:02.503177 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:02.503636 (+ 459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:02.506236 (+ 2600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.506240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.506242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.506558 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.506908 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.506911 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.506915 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.507007 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.507523 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:02.507525 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:02.507627 (+ 102us) client_negotiation.cc:769] Sending connection context
0808 18:29:02.508425 (+ 798us) client_negotiation.cc:240] Negotiation successful
0808 18:29:02.508595 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":68,"spinlock_wait_cycles":1280,"thread_start_us":142,"threads_started":1}
I20260808 18:29:02.509567 5194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.468118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.0.0.1:48896)
0808 18:29:02.468546 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.468562 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:02.468654 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:02.468921 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:02.468924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:02.469142 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:02.469318 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.469325 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.470442 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.470445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.472267 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.472278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.473002 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.473005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.473144 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.503877 (+ 30733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:02.503890 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:02.504385 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:02.506658 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.506661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.506662 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.506899 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.507255 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.507258 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.507260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.507336 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.507831 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:02.507834 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:02.507933 (+ 99us) client_negotiation.cc:769] Sending connection context
0808 18:29:02.509224 (+ 1291us) client_negotiation.cc:240] Negotiation successful
0808 18:29:02.509404 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":161,"spinlock_wait_cycles":3328,"thread_start_us":132,"threads_started":1}
Aug 08 18:29:02 dist-test-slave-gp6n krb5kdc[4317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213741, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.190@KRBTEST.COM for oryx/127.29.78.188@KRBTEST.COM
I20260808 18:29:02.521474 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.513715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50274 (local address 127.29.78.188:43147)
0808 18:29:02.513819 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.513824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.513889 (+ 65us) server_negotiation.cc:407] Connection header received
0808 18:29:02.514049 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.514052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.514106 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.514191 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:02.514659 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.515161 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.515961 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.516193 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.516295 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.518277 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:02.518280 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:02.518283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:02.518305 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:02.519775 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.520292 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.520295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.520297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.520383 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.520661 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.520666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.520668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.520847 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:02.520953 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.521207 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.521316 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 18:29:02.521476 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.513643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43147 (local address 127.0.0.1:50274)
0808 18:29:02.513819 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.513834 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:02.513931 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:02.514206 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:02.514209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:02.514397 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:02.514553 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.514559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.515276 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.515281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.515831 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.515846 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.516308 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.516311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.516406 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.516873 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:02.516889 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:02.518144 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:02.519913 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.519916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.519918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.520185 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.520474 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.520478 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.520480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.520570 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.520943 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:02.520946 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:02.521048 (+ 102us) client_negotiation.cc:769] Sending connection context
0808 18:29:02.521156 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 18:29:02.521309 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260808 18:29:02.522802 4366 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.29.78.188:43147: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6"
member_type: NON_VOTER
last_known_addr {
host: "127.29.78.188"
port: 43147
}
attrs {
promote: true
}
}
dest_uuid: "181bd4194326418da33169a4f27346f8"
cas_config_opid_index: 5
I20260808 18:29:02.523378 4366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: true } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: true } }
I20260808 18:29:02.524493 4990 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 12f6a1d683a94e278adf262ca5911ca6 (127.29.78.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: true } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: true } } }
I20260808 18:29:02.525698 4990 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.29.78.188:43147
I20260808 18:29:02.525897 4990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "181bd4194326418da33169a4f27346f8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: true } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: true } } }
I20260808 18:29:02.526014 4990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [sys.catalog]: This master's current role is: LEADER
I20260808 18:29:02.526479 5001 master.cc:537] Master@127.29.78.188:43147 shutting down...
W20260808 18:29:02.527581 4350 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260808 18:29:02.528695 4350 consensus_peers.cc:487] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 -> Peer 12f6a1d683a94e278adf262ca5911ca6 (127.29.78.188:43147): Couldn't send request to peer 12f6a1d683a94e278adf262ca5911ca6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 18:29:02.532131 5001 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260808 18:29:02.532464 5001 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260808 18:29:02.533385 5001 tablet_replica.cc:330] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6: stopping tablet replica
I20260808 18:29:02.535560 5172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.527008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54016 (local address 127.29.78.189:32899)
0808 18:29:02.527218 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.527223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.530721 (+ 3498us) server_negotiation.cc:407] Connection header received
0808 18:29:02.530973 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.530976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.531055 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.531124 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:02.531725 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.532548 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.534041 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.534690 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.534770 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.534909 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.535315 (+ 406us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.535411 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260808 18:29:02.536815 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.526615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:32899 (local address 127.0.0.1:54016)
0808 18:29:02.530636 (+ 4021us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.530654 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:02.530800 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:02.531292 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:02.531298 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:02.531321 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:02.531574 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.531585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.532738 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.532741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.533897 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.533908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.535018 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.535024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.535134 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.535187 (+ 53us) client_negotiation.cc:769] Sending connection context
0808 18:29:02.536558 (+ 1371us) client_negotiation.cc:240] Negotiation successful
0808 18:29:02.536660 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3928}
I20260808 18:29:02.537755 5049 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 1 LEARNER]: Refusing update from remote peer 181bd4194326418da33169a4f27346f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 18:29:02.538321 4989 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 18:29:02.539288 5001 master.cc:552] Master@127.29.78.188:43147 shutdown complete.
I20260808 18:29:02.539369 5001 master_runner.cc:305] Clearing existing system tablet
I20260808 18:29:02.539640 5001 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:02.539728 5001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:02.539778 5049 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 12f6a1d683a94e278adf262ca5911ca6 (127.29.78.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: true } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: true } } }
I20260808 18:29:02.541218 5001 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 18:29:02.541635 5082 sys_catalog.cc:455] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 181bd4194326418da33169a4f27346f8. Latest consensus state: current_term: 1 leader_uuid: "181bd4194326418da33169a4f27346f8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: true } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: true } } }
I20260808 18:29:02.541738 5082 sys_catalog.cc:458] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: This master's current role is: LEARNER
I20260808 18:29:02.541837 5082 sys_catalog.cc:455] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "181bd4194326418da33169a4f27346f8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: true } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: true } } }
I20260808 18:29:02.541929 5082 sys_catalog.cc:458] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: This master's current role is: LEARNER
I20260808 18:29:02.542258 5209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:02.542865 5001 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:02.543012 5001 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 18:29:02.543051 5001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/wal
uuid: "12f6a1d683a94e278adf262ca5911ca6"
format_stamp: "Formatted at 2026-08-08 18:29:02 on dist-test-slave-gp6n"
server_key: "5a2de8bbfa1a2cf42ebe42a26238ddfa"
server_key_iv: "af4cdf8bdbf4b2f84f9459ea4ddcd582"
server_key_version: "encryptionkey@0"
I20260808 18:29:02.543301 4989 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8: attempting to promote NON_VOTER a692e2279f124613a8b62dc5c5cda6ce to VOTER
I20260808 18:29:02.543728 4989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: true } }
I20260808 18:29:02.545068 5001 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:29:02.545707 5049 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 1 LEARNER]: Refusing update from remote peer 181bd4194326418da33169a4f27346f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 18:29:02.546146 4990 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 18:29:02.547437 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.545256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43147 (local address 127.0.0.1:50284)
0808 18:29:02.547259 (+ 2003us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.547343 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1908}
I20260808 18:29:02.547583 5001 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 18:29:02.547672 5001 log.cc:1196] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260808 18:29:02.547936 5001 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6: Deleting consensus metadata
I20260808 18:29:02.548130 5001 master_runner.cc:315] Copying system tablet from 127.29.78.190:34937
I20260808 18:29:02.548475 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.548179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43147 (local address 127.0.0.1:50294)
0808 18:29:02.548364 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.548416 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260808 18:29:02.548667 4350 consensus_peers.cc:487] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 -> Peer 12f6a1d683a94e278adf262ca5911ca6 (127.29.78.188:43147): Couldn't send request to peer 12f6a1d683a94e278adf262ca5911ca6. Status: Network error: Client connection negotiation failed: client connection to 127.29.78.188:43147: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 18:29:02.549379 5001 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6: tablet copy: Beginning tablet copy session from remote peer at address 127.29.78.190:34937
I20260808 18:29:02.549593 4990 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a692e2279f124613a8b62dc5c5cda6ce (127.29.78.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: true } } }
I20260808 18:29:02.550920 4990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "181bd4194326418da33169a4f27346f8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: true } } }
I20260808 18:29:02.551025 4990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [sys.catalog]: This master's current role is: LEADER
I20260808 18:29:02.551172 4990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "181bd4194326418da33169a4f27346f8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: true } } }
I20260808 18:29:02.551245 4990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [sys.catalog]: This master's current role is: LEADER
I20260808 18:29:02.552161 5049 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a692e2279f124613a8b62dc5c5cda6ce (127.29.78.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: true } } }
I20260808 18:29:02.553584 5208 sys_catalog.cc:455] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "181bd4194326418da33169a4f27346f8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: true } } }
I20260808 18:29:02.553704 5208 sys_catalog.cc:458] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:29:02.562765 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.554898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48912 (local address 127.29.78.190:34937)
0808 18:29:02.555018 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.555022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.555249 (+ 227us) server_negotiation.cc:407] Connection header received
0808 18:29:02.555412 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.555415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.555469 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.555554 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:02.556012 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.556543 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.557570 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.557796 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.557906 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.559261 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:02.559264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:02.559267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:02.559297 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:02.561011 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.561492 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.561496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.561498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.561586 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.561877 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.561880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.561882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.562067 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:02.562229 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.562499 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.562609 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 18:29:02.562779 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.554856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.0.0.1:48912)
0808 18:29:02.555160 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.555172 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:02.555299 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:02.555575 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:02.555578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:02.555792 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:02.555897 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.555902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.556657 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.556661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.557454 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.557463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.557909 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.557913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.558009 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.558702 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:02.558723 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:02.559081 (+ 358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:02.561171 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.561174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.561176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.561383 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.561678 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.561683 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.561685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.561775 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.562166 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:02.562169 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:02.562281 (+ 112us) client_negotiation.cc:769] Sending connection context
0808 18:29:02.562398 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 18:29:02.562580 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":107,"threads_started":1}
I20260808 18:29:02.563529 4387 tablet_copy_service.cc:140] P 181bd4194326418da33169a4f27346f8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 12f6a1d683a94e278adf262ca5911ca6 ({username='oryx', principal='oryx/127.29.78.188@KRBTEST.COM'} at 127.0.0.1:48912)
I20260808 18:29:02.563642 4387 tablet_copy_service.cc:161] P 181bd4194326418da33169a4f27346f8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 12f6a1d683a94e278adf262ca5911ca6 at {username='oryx', principal='oryx/127.29.78.188@KRBTEST.COM'} at 127.0.0.1:48912: session id = 12f6a1d683a94e278adf262ca5911ca6-00000000000000000000000000000000
I20260808 18:29:02.564448 4387 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 18:29:02.565621 5001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 18:29:02.567909 5001 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6: tablet copy: Starting download of 0 data blocks...
I20260808 18:29:02.568120 5001 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6: tablet copy: Starting download of 1 WAL segments...
I20260808 18:29:02.569456 5001 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 18:29:02.570910 4387 tablet_copy_service.cc:342] P 181bd4194326418da33169a4f27346f8: Request end of tablet copy session 12f6a1d683a94e278adf262ca5911ca6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.29.78.188@KRBTEST.COM'} at 127.0.0.1:48912
I20260808 18:29:02.571059 4387 tablet_copy_service.cc:433] P 181bd4194326418da33169a4f27346f8: ending tablet copy session 12f6a1d683a94e278adf262ca5911ca6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 12f6a1d683a94e278adf262ca5911ca6
I20260808 18:29:02.577309 5001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:02.579015 5217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:02.579123 5220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:02.579319 5001 server_base.cc:1034] running on GCE node
W20260808 18:29:02.582580 5218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:02.582798 5001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:02.583127 5001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:02.584270 5001 hybrid_clock.cc:648] HybridClock initialized: now 1786213742584254 us; error 35 us; skew 500 ppm
Aug 08 18:29:02 dist-test-slave-gp6n krb5kdc[4317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213742, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:02.585819 5001 init.cc:377] Logged in from keytab as oryx/127.29.78.188@KRBTEST.COM (short username oryx)
I20260808 18:29:02.586277 5001 webserver.cc:460] Webserver started at http://127.29.78.188:38241/ using document root <none> and password file <none>
I20260808 18:29:02.586524 5001 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:02.586575 5001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:02.587685 5001 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 18:29:02.588608 5227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:02.588833 5001 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 18:29:02.588929 5001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/wal
uuid: "12f6a1d683a94e278adf262ca5911ca6"
format_stamp: "Formatted at 2026-08-08 18:29:02 on dist-test-slave-gp6n"
server_key: "5a2de8bbfa1a2cf42ebe42a26238ddfa"
server_key_iv: "af4cdf8bdbf4b2f84f9459ea4ddcd582"
server_key_version: "encryptionkey@0"
I20260808 18:29:02.589030 5001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:02.613866 5001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:02.615919 5001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:02.620987 5001 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.188:43147
I20260808 18:29:02.621011 5282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.188:43147 every 8 connection(s)
I20260808 18:29:02.621600 5001 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/master-2/data/info.pb
I20260808 18:29:02.623212 5283 sys_catalog.cc:263] Verifying existing consensus state
I20260808 18:29:02.624033 5283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6: Bootstrap starting.
I20260808 18:29:02.633963 5283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:29:02.634246 5283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6: Bootstrap complete.
I20260808 18:29:02.634763 5283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: true } }
I20260808 18:29:02.634902 5283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 12f6a1d683a94e278adf262ca5911ca6, State: Initialized, Role: LEARNER
I20260808 18:29:02.635020 5283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: true } }
I20260808 18:29:02.635360 5286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: true } } }
I20260808 18:29:02.635526 5283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:29:02.635535 5286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [sys.catalog]: This master's current role is: LEARNER
I20260808 18:29:02.639974 5301 catalog_manager.cc:1259] Loaded cluster ID: 79b2d9f0f7a04df6b2088bd4ca8342eb
I20260808 18:29:02.640051 5301 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6: loading cluster ID for follower catalog manager: success
I20260808 18:29:02.641527 5301 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6: acquiring CA information for follower catalog manager: success
I20260808 18:29:02.642760 5301 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 18:29:02.643126 5204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:02.643437 5204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:02.643501 5204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:02.649345 5204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:02.649418 5204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:02.649504 5204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:29:02.650030 4722 heartbeater.cc:503] Master 127.29.78.190:34937 was elected leader, sending a full tablet report...
I20260808 18:29:02.657253 5204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.129.keytab
--principal=oryx/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:46615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=39361
--tserver_master_addrs=127.29.78.190:34937,127.29.78.189:32899,127.29.78.188:43147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:02.658558 5204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:02.659912 5204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:02.671319 5304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:02.671466 5306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:02.671342 5303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:02.671705 5204 server_base.cc:1034] running on GCE node
I20260808 18:29:02.672197 5204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:02.672899 5204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:02.674095 5204 hybrid_clock.cc:648] HybridClock initialized: now 1786213742674075 us; error 38 us; skew 500 ppm
Aug 08 18:29:02 dist-test-slave-gp6n krb5kdc[4317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213742, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:02.677260 5204 init.cc:377] Logged in from keytab as oryx/127.29.78.129@KRBTEST.COM (short username oryx)
I20260808 18:29:02.678344 5204 webserver.cc:460] Webserver started at http://127.29.78.129:39361/ using document root <none> and password file <none>
I20260808 18:29:02.678967 5204 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:02.679080 5204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:02.683281 5204 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:29:02.685382 5313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:02.686638 5204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:29:02.686774 5204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "01781749e0d0467da710bf3c66b32f88"
format_stamp: "Formatted at 2026-08-08 18:29:01 on dist-test-slave-gp6n"
server_key: "ad1b0d69a566dcedbdc27a2ca009ba1d"
server_key_iv: "fc60abd8e2a8ce2a5528087a9e4f4779"
server_key_version: "encryptionkey@0"
I20260808 18:29:02.687327 5204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:02.719383 5204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:02.722801 5204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:02.723050 5204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:02.723690 5204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:29:02.724556 5204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:02.724627 5204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:02.724700 5204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:02.724742 5204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:02.734153 5204 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:46615
I20260808 18:29:02.734191 5435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:46615 every 8 connection(s)
I20260808 18:29:02.735155 5204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:29:02.743508 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 5204
I20260808 18:29:02.743706 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 4583
Aug 08 18:29:02 dist-test-slave-gp6n krb5kdc[4317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213742, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260808 18:29:02.752544 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.737039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:46531 (local address 127.29.78.190:34937)
0808 18:29:02.737183 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.737188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.738385 (+ 1197us) server_negotiation.cc:407] Connection header received
0808 18:29:02.739646 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.739650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.739710 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.739765 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:02.741150 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.741956 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.743184 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.743505 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.743589 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.747667 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:02.747674 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:02.747680 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:02.747712 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:02.749481 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.750272 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.750278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.750282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.750370 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.750856 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.750862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.750867 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.751092 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:02.751183 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.752078 (+ 895us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.752255 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 18:29:02.754980 5442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.737512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.29.78.129:46531)
0808 18:29:02.738191 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.738256 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:02.739405 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:02.739957 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:02.739964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:02.740432 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:02.740999 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.741008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.742095 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.742099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.743047 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.743056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.743781 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.743786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.744012 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.745060 (+ 1048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:02.745082 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:02.747478 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:02.749637 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.749647 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.749668 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.750153 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.750487 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.750493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.750498 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.750742 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.751209 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:02.751219 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:02.751655 (+ 436us) client_negotiation.cc:769] Sending connection context
0808 18:29:02.752770 (+ 1115us) client_negotiation.cc:240] Negotiation successful
0808 18:29:02.753231 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":70,"spinlock_wait_cycles":32768,"thread_start_us":154,"threads_started":1}
Aug 08 18:29:02 dist-test-slave-gp6n krb5kdc[4317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213742, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.188@KRBTEST.COM
I20260808 18:29:02.756143 5437 heartbeater.cc:347] Connected to a master server at 127.29.78.190:34937
I20260808 18:29:02.756349 5437 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:02.756887 5437 heartbeater.cc:511] Master 127.29.78.190:34937 requested a full tablet report, sending...
I20260808 18:29:02.758087 4366 ts_manager.cc:194] Re-registered known tserver with Master: 01781749e0d0467da710bf3c66b32f88 (127.29.78.129:46615)
I20260808 18:29:02.758836 4366 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:46531
I20260808 18:29:02.760193 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.737512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43147 (local address 127.29.78.129:55557)
0808 18:29:02.738483 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.738495 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:02.739405 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:02.739980 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:02.739982 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:02.740437 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:02.741000 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.741008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.742340 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.742343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.743200 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.743209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.743863 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.743866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.753791 (+ 9925us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.754272 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:02.754283 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:02.755901 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:02.758148 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.758154 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.758159 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.758455 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.758794 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.758800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.758805 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.758902 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.759468 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:02.759474 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:02.759583 (+ 109us) client_negotiation.cc:769] Sending connection context
0808 18:29:02.759683 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 18:29:02.759860 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":9960,"spinlock_wait_cycles":53888,"thread_start_us":129,"threads_started":1}
I20260808 18:29:02.760845 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.737258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:55557 (local address 127.29.78.188:43147)
0808 18:29:02.737516 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.737537 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.738564 (+ 1027us) server_negotiation.cc:407] Connection header received
0808 18:29:02.739722 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.739725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.739780 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.739862 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:02.741150 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.742215 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.743313 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.743652 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.743727 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.756095 (+ 12368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:02.756101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:02.756107 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:02.756143 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:02.758000 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.758577 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.758583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.758588 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.758681 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.759019 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.759025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.759040 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.759301 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:02.759400 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.760442 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.760595 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":80,"threads_started":1}
I20260808 18:29:02.761513 5436 heartbeater.cc:347] Connected to a master server at 127.29.78.188:43147
I20260808 18:29:02.761620 5436 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:02.761811 5436 heartbeater.cc:511] Master 127.29.78.188:43147 requested a full tablet report, sending...
I20260808 18:29:02.762221 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:38341
--local_ip_for_outbound_sockets=127.29.78.130
--tserver_master_addrs=127.29.78.190:34937
--webserver_port=33869
--webserver_interface=127.29.78.130
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.130.keytab
--principal=oryx/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.78.190:34937,127.29.78.189:32899
--tserver_master_addrs=127.29.78.190:34937,127.29.78.189:32899,127.29.78.188:43147 with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:29:02.762935 5245 ts_manager.cc:194] Registered new tserver with Master: 01781749e0d0467da710bf3c66b32f88 (127.29.78.129:46615)
Aug 08 18:29:02 dist-test-slave-gp6n krb5kdc[4317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213742, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.189@KRBTEST.COM
I20260808 18:29:02.767586 5444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.738733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:32899 (local address 127.29.78.129:54699)
0808 18:29:02.738985 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.738998 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:02.739424 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:02.739956 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:02.739962 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:02.740431 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:02.740988 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.741002 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.741815 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.741819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.742781 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.742790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.761724 (+ 18934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.761731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.761845 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.762300 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:02.762315 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:02.763936 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:02.765849 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.765851 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.765853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.766041 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.766359 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.766361 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.766362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.766510 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.766955 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:02.766957 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:02.767029 (+ 72us) client_negotiation.cc:769] Sending connection context
0808 18:29:02.767100 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 18:29:02.767214 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":53,"thread_start_us":91,"threads_started":1}
I20260808 18:29:02.767599 5172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.739004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:54699 (local address 127.29.78.189:32899)
0808 18:29:02.739135 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.739139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.739163 (+ 24us) server_negotiation.cc:407] Connection header received
0808 18:29:02.739625 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.739628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.739690 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.739768 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:02.741131 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.741670 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.742904 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.743144 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.743258 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.764139 (+ 20881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:02.764147 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:02.764152 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:02.764192 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:02.765743 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.766153 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.766160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.766165 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.766260 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.766587 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.766593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.766598 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.766854 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:02.766965 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.767215 (+ 250us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.767370 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 18:29:02.768384 5438 heartbeater.cc:347] Connected to a master server at 127.29.78.189:32899
I20260808 18:29:02.768486 5438 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:02.768661 5438 heartbeater.cc:511] Master 127.29.78.189:32899 requested a full tablet report, sending...
I20260808 18:29:02.769707 5031 ts_manager.cc:194] Registered new tserver with Master: 01781749e0d0467da710bf3c66b32f88 (127.29.78.129:46615)
I20260808 18:29:02.855260 4879 heartbeater.cc:503] Master 127.29.78.190:34937 was elected leader, sending a full tablet report...
W20260808 18:29:02.878597 5445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:02.878827 5445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:02.878899 5445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:02.882985 5445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:02.883039 5445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:02.883131 5445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:29:02.888177 5445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.130.keytab
--principal=oryx/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:38341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=33869
--tserver_master_addrs=127.29.78.190:34937,127.29.78.189:32899,127.29.78.188:43147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:02.889191 5445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:02.890352 5445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:02.899978 5450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:02.900039 5453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:02.900161 5445 server_base.cc:1034] running on GCE node
W20260808 18:29:02.900029 5451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:02.900652 5445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:02.901309 5445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:02.902479 5445 hybrid_clock.cc:648] HybridClock initialized: now 1786213742902459 us; error 29 us; skew 500 ppm
Aug 08 18:29:02 dist-test-slave-gp6n krb5kdc[4317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213742, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:02.905205 5445 init.cc:377] Logged in from keytab as oryx/127.29.78.130@KRBTEST.COM (short username oryx)
I20260808 18:29:02.906044 5445 webserver.cc:460] Webserver started at http://127.29.78.130:33869/ using document root <none> and password file <none>
I20260808 18:29:02.906615 5445 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:02.906689 5445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:02.910355 5445 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:29:02.912531 5461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:02.913689 5445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:29:02.913803 5445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "51548778d14f48d49953bdb29879b2ca"
format_stamp: "Formatted at 2026-08-08 18:29:01 on dist-test-slave-gp6n"
server_key: "6eac9c78bffb21351ca6ecdddbfc8bcd"
server_key_iv: "4527aea866c3a722da5a5fef120d2966"
server_key_version: "encryptionkey@0"
I20260808 18:29:02.914320 5445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:02.926539 5445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:02.929149 5445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:02.929363 5445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:02.929932 5445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:29:02.930853 5445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:02.930897 5445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:02.930964 5445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:02.930997 5445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:02.940671 5445 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:38341
I20260808 18:29:02.940692 5587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:38341 every 8 connection(s)
I20260808 18:29:02.941764 5445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:29:02.951283 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 5445
I20260808 18:29:02.951488 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 4728
Aug 08 18:29:02 dist-test-slave-gp6n krb5kdc[4317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213742, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.188@KRBTEST.COM
Aug 08 18:29:02 dist-test-slave-gp6n krb5kdc[4317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213742, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.189@KRBTEST.COM
Aug 08 18:29:02 dist-test-slave-gp6n krb5kdc[4317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213742, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260808 18:29:02.958015 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.945352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:44891 (local address 127.29.78.188:43147)
0808 18:29:02.945491 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.945495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.945593 (+ 98us) server_negotiation.cc:407] Connection header received
0808 18:29:02.946182 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.946186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.946247 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.946324 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:02.947825 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.948655 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.949749 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.950042 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.950144 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.953346 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:02.953349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:02.953352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:02.953376 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:02.955428 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.956091 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.956095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.956097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.956223 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.956684 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.956687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.956689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.956886 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:02.956999 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.957654 (+ 655us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.957800 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 18:29:02.959836 5594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.945246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43147 (local address 127.29.78.130:44891)
0808 18:29:02.945519 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.945534 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:02.945966 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:02.946555 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:02.946561 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:02.947069 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:02.947673 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.947687 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.948811 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.948814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.949626 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.949635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.950191 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.950195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.950379 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.951001 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:02.951016 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:02.953170 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:02.955569 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.955577 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.955593 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.955980 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.956352 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.956357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.956362 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.956528 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.956998 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:02.957005 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:02.957340 (+ 335us) client_negotiation.cc:769] Sending connection context
0808 18:29:02.958202 (+ 862us) client_negotiation.cc:240] Negotiation successful
0808 18:29:02.958590 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":145,"thread_start_us":91,"threads_started":1}
I20260808 18:29:02.960517 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.943644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:35143 (local address 127.29.78.190:34937)
0808 18:29:02.943859 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.943862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.944978 (+ 1116us) server_negotiation.cc:407] Connection header received
0808 18:29:02.946193 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.946198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.946256 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.946309 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:02.947858 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.948650 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.949990 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.950315 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.950450 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.956119 (+ 5669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:02.956134 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:02.956137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:02.956169 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:02.958042 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.958863 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.958866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.958868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.958957 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.959274 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.959278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.959280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.959493 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:02.959578 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.960272 (+ 694us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.960405 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260808 18:29:02.960989 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.944135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.29.78.130:35143)
0808 18:29:02.944804 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.944868 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:02.945967 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:02.946556 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:02.946563 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:02.947093 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:02.947685 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.947691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.948769 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.948773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.949858 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.949869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.950451 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.950455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.950550 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.951190 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:02.951203 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:02.955945 (+ 4742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:02.958388 (+ 2443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.958391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.958393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.958687 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.959076 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.959078 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.959080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.959171 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.959634 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:02.959637 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:02.959716 (+ 79us) client_negotiation.cc:769] Sending connection context
0808 18:29:02.960738 (+ 1022us) client_negotiation.cc:240] Negotiation successful
0808 18:29:02.960872 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":184,"spinlock_wait_cycles":46592,"thread_start_us":148,"threads_started":1}
I20260808 18:29:02.961015 5172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.943639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:42859 (local address 127.29.78.189:32899)
0808 18:29:02.943772 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:02.943776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:02.945067 (+ 1291us) server_negotiation.cc:407] Connection header received
0808 18:29:02.946275 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:02.946280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:02.946336 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:02.946436 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:02.947856 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.948375 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.950002 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.950312 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.950386 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.953977 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:02.953985 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:02.953991 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:02.954028 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:02.956230 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.957645 (+ 1415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.957652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.957657 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.957767 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:02.958169 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:02.958176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:02.958181 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:02.958525 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:02.958656 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:02.960602 (+ 1946us) server_negotiation.cc:299] Negotiation successful
0808 18:29:02.960766 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 18:29:02.961308 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:02.944135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:32899 (local address 127.29.78.130:42859)
0808 18:29:02.944977 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:02.944991 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:02.945967 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:02.946556 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:02.946563 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:02.947093 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:02.947682 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.947691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.948520 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.948529 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.949858 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:02.949869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:02.950583 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:02.950588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:02.950700 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:02.951363 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:02.951381 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:02.953418 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:02.956383 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.956390 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.957127 (+ 737us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.957483 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.957903 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:02.957909 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:02.957914 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:02.958032 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:02.958714 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:02.958720 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:02.958857 (+ 137us) client_negotiation.cc:769] Sending connection context
0808 18:29:02.958993 (+ 136us) client_negotiation.cc:240] Negotiation successful
0808 18:29:02.959176 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":197,"spinlock_wait_cycles":70400,"thread_start_us":149,"threads_started":1}
I20260808 18:29:02.962248 5590 heartbeater.cc:347] Connected to a master server at 127.29.78.189:32899
I20260808 18:29:02.962531 5590 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:02.963045 5590 heartbeater.cc:511] Master 127.29.78.189:32899 requested a full tablet report, sending...
I20260808 18:29:02.964125 5031 ts_manager.cc:194] Registered new tserver with Master: 51548778d14f48d49953bdb29879b2ca (127.29.78.130:38341)
I20260808 18:29:02.967135 5588 heartbeater.cc:347] Connected to a master server at 127.29.78.188:43147
I20260808 18:29:02.967257 5588 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:02.967451 5588 heartbeater.cc:511] Master 127.29.78.188:43147 requested a full tablet report, sending...
I20260808 18:29:02.968245 5245 ts_manager.cc:194] Registered new tserver with Master: 51548778d14f48d49953bdb29879b2ca (127.29.78.130:38341)
I20260808 18:29:02.968457 5589 heartbeater.cc:347] Connected to a master server at 127.29.78.190:34937
I20260808 18:29:02.968552 5589 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:02.968748 5589 heartbeater.cc:511] Master 127.29.78.190:34937 requested a full tablet report, sending...
I20260808 18:29:02.969408 4366 ts_manager.cc:194] Re-registered known tserver with Master: 51548778d14f48d49953bdb29879b2ca (127.29.78.130:38341)
I20260808 18:29:02.969997 4366 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:35143
I20260808 18:29:02.971261 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:41395
--local_ip_for_outbound_sockets=127.29.78.131
--tserver_master_addrs=127.29.78.190:34937
--webserver_port=45987
--webserver_interface=127.29.78.131
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.131.keytab
--principal=oryx/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.78.190:34937,127.29.78.189:32899
--tserver_master_addrs=127.29.78.190:34937,127.29.78.189:32899,127.29.78.188:43147 with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:29:03.038074 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.033804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43147 (local address 127.0.0.1:50310)
0808 18:29:03.034017 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:03.034035 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:03.034166 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:03.034548 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:03.034551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:03.034572 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:03.034689 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:03.034704 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.035627 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.035632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:03.036682 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:03.036716 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.037608 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.037612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:03.037711 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.037736 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 18:29:03.037827 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 18:29:03.037927 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260808 18:29:03.038075 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.033877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50310 (local address 127.29.78.188:43147)
0808 18:29:03.034019 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:03.034025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:03.034119 (+ 94us) server_negotiation.cc:407] Connection header received
0808 18:29:03.034306 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:03.034309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:03.034378 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:03.034530 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:03.034829 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.035493 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.036836 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.037461 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.037588 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.037758 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:03.037838 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 18:29:03.037916 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 18:29:03.038944 5257 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260808 18:29:03.039364 5286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 181bd4194326418da33169a4f27346f8. Latest consensus state: current_term: 1 leader_uuid: "181bd4194326418da33169a4f27346f8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: true } } }
I20260808 18:29:03.039515 5286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [sys.catalog]: This master's current role is: LEARNER
W20260808 18:29:03.085042 5595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:03.085341 5595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:03.085423 5595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:03.089665 5595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:03.089747 5595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:03.089877 5595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:29:03.095350 5595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.131.keytab
--principal=oryx/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:41395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=45987
--tserver_master_addrs=127.29.78.190:34937,127.29.78.189:32899,127.29.78.188:43147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:03.096453 5595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:03.097532 5595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:03.107232 5602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:03.107211 5604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:03.107615 5595 server_base.cc:1034] running on GCE node
W20260808 18:29:03.107244 5601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:03.108078 5595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:03.108681 5595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:03.111937 5595 hybrid_clock.cc:648] HybridClock initialized: now 1786213743111914 us; error 32 us; skew 500 ppm
Aug 08 18:29:03 dist-test-slave-gp6n krb5kdc[4317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213743, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:03.114866 5595 init.cc:377] Logged in from keytab as oryx/127.29.78.131@KRBTEST.COM (short username oryx)
I20260808 18:29:03.115641 5595 webserver.cc:460] Webserver started at http://127.29.78.131:45987/ using document root <none> and password file <none>
I20260808 18:29:03.116161 5595 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:03.116235 5595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:03.120199 5595 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:29:03.122138 5611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:03.123345 5595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:29:03.123487 5595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "5d47d6d7c4d445dbaebedcaae667f1de"
format_stamp: "Formatted at 2026-08-08 18:29:01 on dist-test-slave-gp6n"
server_key: "f38c32d14a1dffb89710779662107fc9"
server_key_iv: "87d36d20de401b10d61752381eca8901"
server_key_version: "encryptionkey@0"
I20260808 18:29:03.124017 5595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:03.138041 5595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:03.140920 5595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:03.141111 5595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:03.141662 5595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:29:03.142549 5595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:03.142614 5595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:03.142676 5595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:03.142725 5595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:03.152257 5595 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:41395
I20260808 18:29:03.152282 5735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:41395 every 8 connection(s)
I20260808 18:29:03.153227 5595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:29:03.158311 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 5595
Aug 08 18:29:03 dist-test-slave-gp6n krb5kdc[4317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213743, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.189@KRBTEST.COM
Aug 08 18:29:03 dist-test-slave-gp6n krb5kdc[4317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213743, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.188@KRBTEST.COM
Aug 08 18:29:03 dist-test-slave-gp6n krb5kdc[4317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213743, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260808 18:29:03.171481 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.154980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:51141 (local address 127.29.78.188:43147)
0808 18:29:03.155138 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:03.155143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:03.156138 (+ 995us) server_negotiation.cc:407] Connection header received
0808 18:29:03.157103 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:03.157107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:03.157167 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:03.157248 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:03.158725 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.159543 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.161355 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.161706 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.161798 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.166803 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:03.166807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:03.166810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:03.166838 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:03.168780 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:03.169469 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:03.169475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:03.169480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:03.169577 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:03.170064 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:03.170067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:03.170069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:03.170303 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:03.170383 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:03.171165 (+ 782us) server_negotiation.cc:299] Negotiation successful
0808 18:29:03.171291 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67,"spinlock_wait_cycles":1920}
I20260808 18:29:03.172768 5172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.156644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:49595 (local address 127.29.78.189:32899)
0808 18:29:03.156796 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:03.156802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:03.156818 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:29:03.157643 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:03.157647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:03.157696 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:03.157784 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:03.158725 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.159546 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.161288 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.161648 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.161729 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.167126 (+ 5397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:03.167130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:03.167133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:03.167155 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:03.169134 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:03.170712 (+ 1578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:03.170715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:03.170717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:03.170795 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:03.171850 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:03.171853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:03.171855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:03.172049 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:03.172130 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:03.172463 (+ 333us) server_negotiation.cc:299] Negotiation successful
0808 18:29:03.172601 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 18:29:03.173034 5748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.160573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48922 (local address 127.29.78.190:34937)
0808 18:29:03.161172 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:03.161176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:03.161191 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:29:03.161232 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:03.161236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:03.161289 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:03.161366 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:03.162815 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.163639 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.164934 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.165280 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.165390 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.168163 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:03.168170 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:03.168173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:03.168205 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:03.170487 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:03.170981 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:03.170984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:03.170986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:03.171082 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:03.171399 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:03.171403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:03.171404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:03.171911 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:03.172674 (+ 763us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:03.172753 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 18:29:03.172875 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":76,"threads_started":1}
I20260808 18:29:03.173322 5742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.156273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:32899 (local address 127.29.78.131:49595)
0808 18:29:03.156698 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:03.156714 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:03.156904 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:03.157818 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:03.157821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:03.157980 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:03.158553 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:03.158573 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.159717 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.159720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:03.161155 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:03.161165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.161933 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.161937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:03.162156 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.163139 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:03.163159 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:03.166966 (+ 3807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:03.169433 (+ 2467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:03.169439 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:03.169444 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:03.169703 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:03.171653 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:03.171658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:03.171660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:03.171749 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:03.172188 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:03.172191 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:03.172282 (+ 91us) client_negotiation.cc:769] Sending connection context
0808 18:29:03.172967 (+ 685us) client_negotiation.cc:240] Negotiation successful
0808 18:29:03.173123 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":88,"spinlock_wait_cycles":2944,"thread_start_us":85,"threads_started":1}
I20260808 18:29:03.173540 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.155341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43147 (local address 127.29.78.131:51141)
0808 18:29:03.156038 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:03.156053 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:03.156902 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:03.157383 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:03.157389 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:03.157789 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:03.158553 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:03.158573 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.159717 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.159720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:03.161207 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:03.161218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.161972 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.161975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:03.162152 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.163341 (+ 1189us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:03.163361 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:03.166215 (+ 2854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:03.168936 (+ 2721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:03.168943 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:03.168960 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:03.169343 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:03.169711 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:03.169715 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:03.169719 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:03.169936 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:03.170503 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:03.170511 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:03.170847 (+ 336us) client_negotiation.cc:769] Sending connection context
0808 18:29:03.171723 (+ 876us) client_negotiation.cc:240] Negotiation successful
0808 18:29:03.172103 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":341,"spinlock_wait_cycles":51584,"thread_start_us":175,"threads_started":1}
I20260808 18:29:03.174532 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.161038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54028 (local address 127.29.78.189:32899)
0808 18:29:03.161952 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:03.161956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:03.162320 (+ 364us) server_negotiation.cc:407] Connection header received
0808 18:29:03.162364 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:03.162366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:03.162452 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:03.162538 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:03.163463 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.164242 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.165625 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.165993 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.166100 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.167857 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:03.167866 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:03.167872 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:03.167907 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:03.170491 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:03.171187 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:03.171192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:03.171194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:03.171281 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:03.172042 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:03.172045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:03.172047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:03.172231 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:03.172358 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:03.174163 (+ 1805us) server_negotiation.cc:299] Negotiation successful
0808 18:29:03.174289 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":71,"threads_started":1}
I20260808 18:29:03.176057 5736 heartbeater.cc:347] Connected to a master server at 127.29.78.188:43147
I20260808 18:29:03.176311 5736 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:03.176939 5736 heartbeater.cc:511] Master 127.29.78.188:43147 requested a full tablet report, sending...
I20260808 18:29:03.177510 5738 heartbeater.cc:347] Connected to a master server at 127.29.78.189:32899
I20260808 18:29:03.177590 5738 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:03.177937 5738 heartbeater.cc:511] Master 127.29.78.189:32899 requested a full tablet report, sending...
I20260808 18:29:03.178158 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.156313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:46585 (local address 127.29.78.190:34937)
0808 18:29:03.156454 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:03.156461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:03.156480 (+ 19us) server_negotiation.cc:407] Connection header received
0808 18:29:03.157363 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:03.157366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:03.157409 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:03.157508 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:03.158733 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.159532 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.163989 (+ 4457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.164337 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.164410 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.174165 (+ 9755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:03.174168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:03.174171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:03.174195 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:03.176157 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:03.176723 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:03.176727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:03.176729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:03.176805 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:03.177167 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:03.177170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:03.177172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:03.177358 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:03.177441 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:03.177898 (+ 457us) server_negotiation.cc:299] Negotiation successful
0808 18:29:03.178018 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:29:03.178848 5741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.155341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.29.78.131:46585)
0808 18:29:03.155891 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:03.155953 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:03.156892 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:03.157523 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:03.157526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:03.157791 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:03.158554 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:03.158572 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.159714 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.159718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:03.163170 (+ 3452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:03.163180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.164591 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.164594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:03.164685 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.165554 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:03.165573 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:03.174024 (+ 8451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:03.176312 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:03.176316 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:03.176317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:03.176602 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:03.176906 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:03.176909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:03.176922 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:03.177061 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:03.177498 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:03.177501 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:03.177718 (+ 217us) client_negotiation.cc:769] Sending connection context
0808 18:29:03.178488 (+ 770us) client_negotiation.cc:240] Negotiation successful
0808 18:29:03.178685 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":1565,"spinlock_wait_cycles":3456,"thread_start_us":109,"threads_started":1}
I20260808 18:29:03.178840 5245 ts_manager.cc:194] Registered new tserver with Master: 5d47d6d7c4d445dbaebedcaae667f1de (127.29.78.131:41395)
I20260808 18:29:03.179374 5031 ts_manager.cc:194] Registered new tserver with Master: 5d47d6d7c4d445dbaebedcaae667f1de (127.29.78.131:41395)
I20260808 18:29:03.179715 5737 heartbeater.cc:347] Connected to a master server at 127.29.78.190:34937
I20260808 18:29:03.179805 5737 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:03.179754 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.161703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50326 (local address 127.29.78.188:43147)
0808 18:29:03.165073 (+ 3370us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:03.165079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:03.165710 (+ 631us) server_negotiation.cc:407] Connection header received
0808 18:29:03.167083 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:03.167088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:03.167140 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:03.167229 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:03.168556 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.169399 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.170624 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.170979 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.171091 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.175724 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:03.175728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:03.175730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:03.175755 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:03.177534 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:03.178033 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:03.178036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:03.178037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:03.178123 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:03.178719 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:03.178722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:03.178723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:03.178935 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:03.179053 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:03.179399 (+ 346us) server_negotiation.cc:299] Negotiation successful
0808 18:29:03.179587 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3273,"thread_start_us":63,"threads_started":1}
I20260808 18:29:03.179970 5737 heartbeater.cc:511] Master 127.29.78.190:34937 requested a full tablet report, sending...
I20260808 18:29:03.176627 4366 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 18:29:03.180759 4365 ts_manager.cc:194] Re-registered known tserver with Master: 5d47d6d7c4d445dbaebedcaae667f1de (127.29.78.131:41395)
I20260808 18:29:03.181299 4365 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:46585
W20260808 18:29:03.181473 4366 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:29:03.184260 5210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "181bd4194326418da33169a4f27346f8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: true } } }
I20260808 18:29:03.184399 5210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [sys.catalog]: This master's current role is: LEADER
I20260808 18:29:03.188526 5210 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8: attempting to promote NON_VOTER 12f6a1d683a94e278adf262ca5911ca6 to VOTER
I20260808 18:29:03.188915 5210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: false } }
I20260808 18:29:03.190166 5257 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [term 1 LEARNER]: Refusing update from remote peer 181bd4194326418da33169a4f27346f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 18:29:03.190201 5049 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 1 FOLLOWER]: Refusing update from remote peer 181bd4194326418da33169a4f27346f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 18:29:03.190747 5210 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 18:29:03.191051 5756 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 18:29:03.192924 5755 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 12f6a1d683a94e278adf262ca5911ca6 (127.29.78.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: false } } }
I20260808 18:29:03.193486 5257 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 12f6a1d683a94e278adf262ca5911ca6 (127.29.78.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: false } } }
I20260808 18:29:03.194351 5210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "181bd4194326418da33169a4f27346f8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: false } } }
I20260808 18:29:03.194482 5210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [sys.catalog]: This master's current role is: LEADER
I20260808 18:29:03.194422 5755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "181bd4194326418da33169a4f27346f8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: false } } }
I20260808 18:29:03.194532 5755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [sys.catalog]: This master's current role is: LEADER
I20260808 18:29:03.194514 5754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "181bd4194326418da33169a4f27346f8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: false } } }
I20260808 18:29:03.194597 5754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8 [sys.catalog]: This master's current role is: LEADER
I20260808 18:29:03.194871 5049 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 12f6a1d683a94e278adf262ca5911ca6 (127.29.78.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: false } } }
I20260808 18:29:03.195292 5286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "181bd4194326418da33169a4f27346f8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: false } } }
I20260808 18:29:03.195408 5286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:29:03.196527 5765 sys_catalog.cc:455] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "181bd4194326418da33169a4f27346f8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: false } } }
I20260808 18:29:03.196657 5765 sys_catalog.cc:458] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:29:03.202914 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.197948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:38341 (local address 127.0.0.1:60064)
0808 18:29:03.198123 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:03.198142 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:03.198230 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:03.199108 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:03.199111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:03.199126 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:03.199235 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:03.199244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.200537 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.200541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:03.201363 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:03.201372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.202520 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.202525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:03.202621 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.202647 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 18:29:03.202701 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 18:29:03.202775 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260808 18:29:03.203500 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.198076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60064 (local address 127.29.78.130:38341)
0808 18:29:03.198570 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:03.198585 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:03.198612 (+ 27us) server_negotiation.cc:407] Connection header received
0808 18:29:03.198694 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:03.198700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:03.198932 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:03.199143 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:03.199348 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.200444 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.201538 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.202308 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.203116 (+ 808us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.203248 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:03.203289 (+ 41us) server_negotiation.cc:299] Negotiation successful
0808 18:29:03.203352 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":110,"threads_started":1}
I20260808 18:29:03.203910 5767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.198140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:46615 (local address 127.0.0.1:40688)
0808 18:29:03.198446 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:03.198460 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:03.198546 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:03.199672 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:03.199675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:03.199695 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:03.199791 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:03.199796 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.201009 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.201012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:03.202195 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:03.202228 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.203512 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.203519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:03.203620 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.203648 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 18:29:03.203715 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 18:29:03.203780 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":106,"threads_started":1}
I20260808 18:29:03.204246 5771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.198909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40688 (local address 127.29.78.129:46615)
0808 18:29:03.199192 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:03.199207 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:03.199227 (+ 20us) server_negotiation.cc:407] Connection header received
0808 18:29:03.199323 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:03.199328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:03.199507 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:03.199697 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:03.199907 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.200843 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.202368 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.203386 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.203490 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.203761 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:03.203813 (+ 52us) server_negotiation.cc:299] Negotiation successful
0808 18:29:03.203904 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":94,"threads_started":1}
I20260808 18:29:03.204831 5768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.198611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:41395 (local address 127.0.0.1:55854)
0808 18:29:03.198835 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:03.198849 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:03.198948 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:03.200370 (+ 1422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:03.200373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:03.200393 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:03.200486 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:03.200492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.201977 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.201980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:03.203131 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:03.203143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.204390 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.204393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:03.204507 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.204530 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 18:29:03.204636 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 18:29:03.204705 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":98,"threads_started":1}
I20260808 18:29:03.205041 5769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.198767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.29.78.131:41395)
0808 18:29:03.199972 (+ 1205us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:03.199984 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:03.200006 (+ 22us) server_negotiation.cc:407] Connection header received
0808 18:29:03.200072 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:03.200077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:03.200227 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:03.200359 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:03.200677 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.201850 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.203276 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.204089 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.204189 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.204321 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:03.204666 (+ 345us) server_negotiation.cc:299] Negotiation successful
0808 18:29:03.204752 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1100,"thread_start_us":106,"threads_started":1}
I20260808 18:29:03.206727 5518 tablet_service.cc:1467] Processing CreateTablet for tablet df3faa5685394d25a81bb0776872b5bf (DEFAULT_TABLE table=test-table [id=a20a17e754284e3bb5ed3b7ca26baa98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:29:03.207109 5364 tablet_service.cc:1467] Processing CreateTablet for tablet df3faa5685394d25a81bb0776872b5bf (DEFAULT_TABLE table=test-table [id=a20a17e754284e3bb5ed3b7ca26baa98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:29:03.207885 5518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df3faa5685394d25a81bb0776872b5bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:03.208092 5364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df3faa5685394d25a81bb0776872b5bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:03.208264 5666 tablet_service.cc:1467] Processing CreateTablet for tablet df3faa5685394d25a81bb0776872b5bf (DEFAULT_TABLE table=test-table [id=a20a17e754284e3bb5ed3b7ca26baa98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:29:03.209079 5666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df3faa5685394d25a81bb0776872b5bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:03.215596 5772 tablet_bootstrap.cc:492] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de: Bootstrap starting.
I20260808 18:29:03.216763 5773 tablet_bootstrap.cc:492] T df3faa5685394d25a81bb0776872b5bf P 01781749e0d0467da710bf3c66b32f88: Bootstrap starting.
I20260808 18:29:03.219053 5773 tablet_bootstrap.cc:654] T df3faa5685394d25a81bb0776872b5bf P 01781749e0d0467da710bf3c66b32f88: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:03.219053 5772 tablet_bootstrap.cc:654] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:03.219952 5773 log.cc:824] T df3faa5685394d25a81bb0776872b5bf P 01781749e0d0467da710bf3c66b32f88: Log is configured to *not* fsync() on all Append() calls
I20260808 18:29:03.219952 5772 log.cc:824] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de: Log is configured to *not* fsync() on all Append() calls
I20260808 18:29:03.220711 5775 tablet_bootstrap.cc:492] T df3faa5685394d25a81bb0776872b5bf P 51548778d14f48d49953bdb29879b2ca: Bootstrap starting.
I20260808 18:29:03.221869 5773 tablet_bootstrap.cc:492] T df3faa5685394d25a81bb0776872b5bf P 01781749e0d0467da710bf3c66b32f88: No bootstrap required, opened a new log
I20260808 18:29:03.222047 5772 tablet_bootstrap.cc:492] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de: No bootstrap required, opened a new log
I20260808 18:29:03.222095 5773 ts_tablet_manager.cc:1397] T df3faa5685394d25a81bb0776872b5bf P 01781749e0d0467da710bf3c66b32f88: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 18:29:03.222258 5772 ts_tablet_manager.cc:1397] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 18:29:03.222895 5775 tablet_bootstrap.cc:654] T df3faa5685394d25a81bb0776872b5bf P 51548778d14f48d49953bdb29879b2ca: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:03.223743 5775 log.cc:824] T df3faa5685394d25a81bb0776872b5bf P 51548778d14f48d49953bdb29879b2ca: Log is configured to *not* fsync() on all Append() calls
I20260808 18:29:03.226106 5775 tablet_bootstrap.cc:492] T df3faa5685394d25a81bb0776872b5bf P 51548778d14f48d49953bdb29879b2ca: No bootstrap required, opened a new log
I20260808 18:29:03.226285 5775 ts_tablet_manager.cc:1397] T df3faa5685394d25a81bb0776872b5bf P 51548778d14f48d49953bdb29879b2ca: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:29:03.226148 5773 raft_consensus.cc:348] T df3faa5685394d25a81bb0776872b5bf P 01781749e0d0467da710bf3c66b32f88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51548778d14f48d49953bdb29879b2ca" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38341 } } peers { permanent_uuid: "01781749e0d0467da710bf3c66b32f88" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46615 } } peers { permanent_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41395 } }
I20260808 18:29:03.226469 5773 raft_consensus.cc:374] T df3faa5685394d25a81bb0776872b5bf P 01781749e0d0467da710bf3c66b32f88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:03.226547 5773 raft_consensus.cc:729] T df3faa5685394d25a81bb0776872b5bf P 01781749e0d0467da710bf3c66b32f88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01781749e0d0467da710bf3c66b32f88, State: Initialized, Role: FOLLOWER
I20260808 18:29:03.226742 5772 raft_consensus.cc:348] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51548778d14f48d49953bdb29879b2ca" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38341 } } peers { permanent_uuid: "01781749e0d0467da710bf3c66b32f88" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46615 } } peers { permanent_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41395 } }
I20260808 18:29:03.226995 5772 raft_consensus.cc:374] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:03.227054 5772 raft_consensus.cc:729] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d47d6d7c4d445dbaebedcaae667f1de, State: Initialized, Role: FOLLOWER
I20260808 18:29:03.227097 5773 consensus_queue.cc:260] T df3faa5685394d25a81bb0776872b5bf P 01781749e0d0467da710bf3c66b32f88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51548778d14f48d49953bdb29879b2ca" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38341 } } peers { permanent_uuid: "01781749e0d0467da710bf3c66b32f88" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46615 } } peers { permanent_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41395 } }
I20260808 18:29:03.227520 5772 consensus_queue.cc:260] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51548778d14f48d49953bdb29879b2ca" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38341 } } peers { permanent_uuid: "01781749e0d0467da710bf3c66b32f88" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46615 } } peers { permanent_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41395 } }
I20260808 18:29:03.227813 5437 heartbeater.cc:503] Master 127.29.78.190:34937 was elected leader, sending a full tablet report...
I20260808 18:29:03.229625 5772 ts_tablet_manager.cc:1428] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260808 18:29:03.230041 5737 heartbeater.cc:503] Master 127.29.78.190:34937 was elected leader, sending a full tablet report...
I20260808 18:29:03.231050 5775 raft_consensus.cc:348] T df3faa5685394d25a81bb0776872b5bf P 51548778d14f48d49953bdb29879b2ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51548778d14f48d49953bdb29879b2ca" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38341 } } peers { permanent_uuid: "01781749e0d0467da710bf3c66b32f88" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46615 } } peers { permanent_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41395 } }
I20260808 18:29:03.231320 5775 raft_consensus.cc:374] T df3faa5685394d25a81bb0776872b5bf P 51548778d14f48d49953bdb29879b2ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:03.231382 5775 raft_consensus.cc:729] T df3faa5685394d25a81bb0776872b5bf P 51548778d14f48d49953bdb29879b2ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51548778d14f48d49953bdb29879b2ca, State: Initialized, Role: FOLLOWER
I20260808 18:29:03.231875 5775 consensus_queue.cc:260] T df3faa5685394d25a81bb0776872b5bf P 51548778d14f48d49953bdb29879b2ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51548778d14f48d49953bdb29879b2ca" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38341 } } peers { permanent_uuid: "01781749e0d0467da710bf3c66b32f88" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46615 } } peers { permanent_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41395 } }
I20260808 18:29:03.232872 5775 ts_tablet_manager.cc:1428] T df3faa5685394d25a81bb0776872b5bf P 51548778d14f48d49953bdb29879b2ca: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260808 18:29:03.233184 5589 heartbeater.cc:503] Master 127.29.78.190:34937 was elected leader, sending a full tablet report...
I20260808 18:29:03.235299 5773 ts_tablet_manager.cc:1428] T df3faa5685394d25a81bb0776872b5bf P 01781749e0d0467da710bf3c66b32f88: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
W20260808 18:29:03.237239 5439 tablet.cc:2369] T df3faa5685394d25a81bb0776872b5bf P 01781749e0d0467da710bf3c66b32f88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:29:03.260214 5780 raft_consensus.cc:482] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:29:03.260432 5780 raft_consensus.cc:504] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51548778d14f48d49953bdb29879b2ca" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38341 } } peers { permanent_uuid: "01781749e0d0467da710bf3c66b32f88" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46615 } } peers { permanent_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41395 } }
I20260808 18:29:03.261530 5780 leader_election.cc:288] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51548778d14f48d49953bdb29879b2ca (127.29.78.130:38341), 01781749e0d0467da710bf3c66b32f88 (127.29.78.129:46615)
I20260808 18:29:03.265379 5771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.261860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:50295 (local address 127.29.78.129:46615)
0808 18:29:03.262003 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:03.262009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:03.262092 (+ 83us) server_negotiation.cc:407] Connection header received
0808 18:29:03.262256 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:03.262259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:03.262300 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:03.262358 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:03.262691 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.263200 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.264357 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.264743 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.264858 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.264939 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:03.265166 (+ 227us) server_negotiation.cc:299] Negotiation successful
0808 18:29:03.265242 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 18:29:03.265405 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.261860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:55239 (local address 127.29.78.130:38341)
0808 18:29:03.261992 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:03.261997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:03.262054 (+ 57us) server_negotiation.cc:407] Connection header received
0808 18:29:03.262282 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:03.262285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:03.262328 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:03.262437 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:03.262721 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.263210 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.264335 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.264759 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.264892 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.264946 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:03.265218 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 18:29:03.265299 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 18:29:03.265401 5742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.261761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:38341 (local address 127.29.78.131:55239)
0808 18:29:03.261975 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:03.261991 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:03.262130 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:03.262466 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:03.262472 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:03.262488 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:03.262585 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:03.262589 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.263333 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.263336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:03.264196 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:03.264205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.264896 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.264900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:03.265032 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.265084 (+ 52us) client_negotiation.cc:769] Sending connection context
0808 18:29:03.265179 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 18:29:03.265247 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":36,"spinlock_wait_cycles":16512}
I20260808 18:29:03.265405 5741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.261761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:46615 (local address 127.29.78.131:50295)
0808 18:29:03.261994 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:03.262011 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:03.262131 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:03.262392 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:03.262397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:03.262433 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:03.262569 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:03.262582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.263307 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.263312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:03.264206 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:03.264216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.264861 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.264865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:03.264977 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.265064 (+ 87us) client_negotiation.cc:769] Sending connection context
0808 18:29:03.265125 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 18:29:03.265221 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":139,"spinlock_wait_cycles":3200}
I20260808 18:29:03.266007 5539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3faa5685394d25a81bb0776872b5bf" candidate_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51548778d14f48d49953bdb29879b2ca" is_pre_election: true
I20260808 18:29:03.266340 5539 raft_consensus.cc:2454] T df3faa5685394d25a81bb0776872b5bf P 51548778d14f48d49953bdb29879b2ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d47d6d7c4d445dbaebedcaae667f1de in term 0.
I20260808 18:29:03.266657 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3faa5685394d25a81bb0776872b5bf" candidate_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01781749e0d0467da710bf3c66b32f88" is_pre_election: true
I20260808 18:29:03.266887 5613 leader_election.cc:302] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51548778d14f48d49953bdb29879b2ca, 5d47d6d7c4d445dbaebedcaae667f1de; no voters:
I20260808 18:29:03.266970 5386 raft_consensus.cc:2454] T df3faa5685394d25a81bb0776872b5bf P 01781749e0d0467da710bf3c66b32f88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d47d6d7c4d445dbaebedcaae667f1de in term 0.
I20260808 18:29:03.267165 5780 raft_consensus.cc:2781] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:29:03.267261 5780 raft_consensus.cc:482] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:29:03.267320 5780 raft_consensus.cc:3037] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:03.268337 5780 raft_consensus.cc:504] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51548778d14f48d49953bdb29879b2ca" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38341 } } peers { permanent_uuid: "01781749e0d0467da710bf3c66b32f88" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46615 } } peers { permanent_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41395 } }
I20260808 18:29:03.268698 5780 leader_election.cc:288] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de [CANDIDATE]: Term 1 election: Requested vote from peers 51548778d14f48d49953bdb29879b2ca (127.29.78.130:38341), 01781749e0d0467da710bf3c66b32f88 (127.29.78.129:46615)
I20260808 18:29:03.269022 5539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3faa5685394d25a81bb0776872b5bf" candidate_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51548778d14f48d49953bdb29879b2ca"
I20260808 18:29:03.269069 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3faa5685394d25a81bb0776872b5bf" candidate_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01781749e0d0467da710bf3c66b32f88"
I20260808 18:29:03.269169 5386 raft_consensus.cc:3037] T df3faa5685394d25a81bb0776872b5bf P 01781749e0d0467da710bf3c66b32f88 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:03.269160 5539 raft_consensus.cc:3037] T df3faa5685394d25a81bb0776872b5bf P 51548778d14f48d49953bdb29879b2ca [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:03.270442 5539 raft_consensus.cc:2454] T df3faa5685394d25a81bb0776872b5bf P 51548778d14f48d49953bdb29879b2ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d47d6d7c4d445dbaebedcaae667f1de in term 1.
I20260808 18:29:03.270442 5386 raft_consensus.cc:2454] T df3faa5685394d25a81bb0776872b5bf P 01781749e0d0467da710bf3c66b32f88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d47d6d7c4d445dbaebedcaae667f1de in term 1.
I20260808 18:29:03.270756 5615 leader_election.cc:302] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01781749e0d0467da710bf3c66b32f88, 5d47d6d7c4d445dbaebedcaae667f1de; no voters:
I20260808 18:29:03.270980 5780 raft_consensus.cc:2781] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:29:03.271205 5780 raft_consensus.cc:686] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de [term 1 LEADER]: Becoming Leader. State: Replica: 5d47d6d7c4d445dbaebedcaae667f1de, State: Running, Role: LEADER
I20260808 18:29:03.271564 5780 consensus_queue.cc:237] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51548778d14f48d49953bdb29879b2ca" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38341 } } peers { permanent_uuid: "01781749e0d0467da710bf3c66b32f88" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46615 } } peers { permanent_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41395 } }
I20260808 18:29:03.275099 4365 catalog_manager.cc:5495] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d47d6d7c4d445dbaebedcaae667f1de (127.29.78.131). New cstate: current_term: 1 leader_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51548778d14f48d49953bdb29879b2ca" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01781749e0d0467da710bf3c66b32f88" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41395 } health_report { overall_health: HEALTHY } } }
I20260808 18:29:03.298637 5769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.294943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55864 (local address 127.29.78.131:41395)
0808 18:29:03.295104 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:03.295108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:03.295123 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:29:03.295256 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:03.295259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:03.295325 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:03.295439 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:29:03.295699 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.296222 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.297100 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.297342 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.297456 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.297788 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:29:03.298200 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:29:03.298334 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:03.298418 (+ 84us) server_negotiation.cc:299] Negotiation successful
0808 18:29:03.298478 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 18:29:03.305274 5539 raft_consensus.cc:1261] T df3faa5685394d25a81bb0776872b5bf P 51548778d14f48d49953bdb29879b2ca [term 1 FOLLOWER]: Refusing update from remote peer 5d47d6d7c4d445dbaebedcaae667f1de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:29:03.305274 5386 raft_consensus.cc:1261] T df3faa5685394d25a81bb0776872b5bf P 01781749e0d0467da710bf3c66b32f88 [term 1 FOLLOWER]: Refusing update from remote peer 5d47d6d7c4d445dbaebedcaae667f1de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:29:03.306070 5783 consensus_queue.cc:1035] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "51548778d14f48d49953bdb29879b2ca" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:29:03.306275 5780 consensus_queue.cc:1035] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "01781749e0d0467da710bf3c66b32f88" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:29:03.314677 5786 mvcc.cc:205] Tried to move back new op lower bound from 7316331492570058752 to 7316331492442824704. Current Snapshot: MvccSnapshot[applied={T|T < 7316331492570058752}]
I20260808 18:29:03.315099 5788 mvcc.cc:205] Tried to move back new op lower bound from 7316331492570058752 to 7316331492442824704. Current Snapshot: MvccSnapshot[applied={T|T < 7316331492570058752}]
I20260808 18:29:03.319695 5787 mvcc.cc:205] Tried to move back new op lower bound from 7316331492570058752 to 7316331492442824704. Current Snapshot: MvccSnapshot[applied={T|T < 7316331492570058752}]
I20260808 18:29:03.325497 4365 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:29:03.325644 4365 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:29:03.330091 4365 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 181bd4194326418da33169a4f27346f8: Sending DeleteTablet for 3 replicas of tablet df3faa5685394d25a81bb0776872b5bf
I20260808 18:29:03.330899 5364 tablet_service.cc:1514] Processing DeleteTablet for tablet df3faa5685394d25a81bb0776872b5bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:29:03 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:40688
I20260808 18:29:03.331080 5666 tablet_service.cc:1514] Processing DeleteTablet for tablet df3faa5685394d25a81bb0776872b5bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:29:03 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:55854
I20260808 18:29:03.331415 5518 tablet_service.cc:1514] Processing DeleteTablet for tablet df3faa5685394d25a81bb0776872b5bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:29:03 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:60064
I20260808 18:29:03.331874 5799 tablet_replica.cc:330] T df3faa5685394d25a81bb0776872b5bf P 51548778d14f48d49953bdb29879b2ca: stopping tablet replica
I20260808 18:29:03.332300 5799 raft_consensus.cc:2229] T df3faa5685394d25a81bb0776872b5bf P 51548778d14f48d49953bdb29879b2ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:29:03.332489 5795 tablet_replica.cc:330] T df3faa5685394d25a81bb0776872b5bf P 01781749e0d0467da710bf3c66b32f88: stopping tablet replica
I20260808 18:29:03.332754 5799 raft_consensus.cc:2258] T df3faa5685394d25a81bb0776872b5bf P 51548778d14f48d49953bdb29879b2ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:29:03.332775 5795 raft_consensus.cc:2229] T df3faa5685394d25a81bb0776872b5bf P 01781749e0d0467da710bf3c66b32f88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:29:03.333037 5795 raft_consensus.cc:2258] T df3faa5685394d25a81bb0776872b5bf P 01781749e0d0467da710bf3c66b32f88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:29:03.333366 5796 tablet_replica.cc:330] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de: stopping tablet replica
I20260808 18:29:03.333747 5796 raft_consensus.cc:2229] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:29:03.334134 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 4332
I20260808 18:29:03.337327 5796 raft_consensus.cc:2258] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:29:03.338145 5799 ts_tablet_manager.cc:1905] T df3faa5685394d25a81bb0776872b5bf P 51548778d14f48d49953bdb29879b2ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:29:03.339645 5796 ts_tablet_manager.cc:1905] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:29:03.341109 5799 ts_tablet_manager.cc:1918] T df3faa5685394d25a81bb0776872b5bf P 51548778d14f48d49953bdb29879b2ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:29:03.341284 5799 log.cc:1196] T df3faa5685394d25a81bb0776872b5bf P 51548778d14f48d49953bdb29879b2ca: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/df3faa5685394d25a81bb0776872b5bf
I20260808 18:29:03.341703 5799 ts_tablet_manager.cc:1939] T df3faa5685394d25a81bb0776872b5bf P 51548778d14f48d49953bdb29879b2ca: Deleting consensus metadata
I20260808 18:29:03.342497 5796 ts_tablet_manager.cc:1918] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:29:03.342600 5796 log.cc:1196] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/df3faa5685394d25a81bb0776872b5bf
I20260808 18:29:03.342933 5796 ts_tablet_manager.cc:1939] T df3faa5685394d25a81bb0776872b5bf P 5d47d6d7c4d445dbaebedcaae667f1de: Deleting consensus metadata
I20260808 18:29:03.343600 5795 ts_tablet_manager.cc:1905] T df3faa5685394d25a81bb0776872b5bf P 01781749e0d0467da710bf3c66b32f88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:29:03.345748 5795 ts_tablet_manager.cc:1918] T df3faa5685394d25a81bb0776872b5bf P 01781749e0d0467da710bf3c66b32f88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:29:03.345858 5795 log.cc:1196] T df3faa5685394d25a81bb0776872b5bf P 01781749e0d0467da710bf3c66b32f88: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-0/wal/wals/df3faa5685394d25a81bb0776872b5bf
W20260808 18:29:03.345959 5463 connection.cc:535] server connection from 127.0.0.1:60064 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 18:29:03.346189 5795 ts_tablet_manager.cc:1939] T df3faa5685394d25a81bb0776872b5bf P 01781749e0d0467da710bf3c66b32f88: Deleting consensus metadata
W20260808 18:29:03.347338 5318 connection.cc:441] server connection from 127.0.0.1:40688 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40688 (request call id 1) could send its response
W20260808 18:29:03.349923 5615 connection.cc:441] server connection from 127.0.0.1:55854 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55854 (request call id 1) could send its response
I20260808 18:29:03.358364 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.333494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.29.78.189:32899)
0808 18:29:03.333757 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:03.333763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:03.348635 (+ 14872us) server_negotiation.cc:407] Connection header received
0808 18:29:03.348794 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:03.348798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:03.349082 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:03.349226 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:03.350862 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.351456 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.352291 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.352541 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.352656 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.354324 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:03.354328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:03.354331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:03.354357 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:03.356362 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:03.356942 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:03.356945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:03.356947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:03.357032 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:03.357335 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:03.357338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:03.357340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:03.357541 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:03.357636 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:03.358027 (+ 391us) server_negotiation.cc:299] Negotiation successful
0808 18:29:03.358157 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260808 18:29:03.365343 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.355523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50332 (local address 127.29.78.188:43147)
0808 18:29:03.355643 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:03.355647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:03.356063 (+ 416us) server_negotiation.cc:407] Connection header received
0808 18:29:03.356232 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:03.356235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:03.356284 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:03.356348 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:03.357318 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.358093 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.359528 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.359883 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.359959 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.361765 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:03.361769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:03.361772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:03.361797 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:03.363599 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:03.364044 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:03.364048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:03.364050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:03.364140 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:03.364454 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:03.364457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:03.364459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:03.364672 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:03.364814 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:03.365051 (+ 237us) server_negotiation.cc:299] Negotiation successful
0808 18:29:03.365174 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 18:29:03.366230 5172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.355287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54042 (local address 127.29.78.189:32899)
0808 18:29:03.355567 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:03.355573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:03.355781 (+ 208us) server_negotiation.cc:407] Connection header received
0808 18:29:03.355858 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:03.355863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:03.355924 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:03.356002 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:03.356946 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.357673 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.360167 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.360486 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.360567 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.362239 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:03.362242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:03.362245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:03.362267 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:03.364323 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:03.364798 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:03.364801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:03.364803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:03.364893 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:03.365229 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:03.365232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:03.365234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:03.365567 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:03.365658 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:03.365957 (+ 299us) server_negotiation.cc:299] Negotiation successful
0808 18:29:03.366070 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260808 18:29:03.370227 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:03.333565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50328 (local address 127.29.78.188:43147)
0808 18:29:03.333694 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:03.333701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:03.359084 (+ 25383us) server_negotiation.cc:407] Connection header received
0808 18:29:03.359245 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:03.359248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:03.359294 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:03.359361 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:03.360211 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.361495 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.362707 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:03.363057 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:03.363154 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:03.367206 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:03.367209 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:03.367211 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:03.367234 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:03.368664 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:03.369073 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:03.369076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:03.369078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:03.369155 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:03.369443 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:03.369446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:03.369447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:03.369608 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:03.369733 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:03.369971 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 18:29:03.370087 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 18:29:04.318682 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.317982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.29.78.131:45989)
0808 18:29:04.318452 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:04.318561 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":87,"threads_started":1}
I20260808 18:29:04.318737 5810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.318094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.29.78.130:38989)
0808 18:29:04.318558 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:04.318641 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":61,"threads_started":1}
W20260808 18:29:04.319065 5589 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:34937 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
W20260808 18:29:04.319257 5737 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:34937 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
I20260808 18:29:04.321436 5811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.321001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.29.78.129:49893)
0808 18:29:04.321232 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:04.321329 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":74,"threads_started":1}
W20260808 18:29:04.321738 5437 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:34937 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
I20260808 18:29:04.830579 5812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 181bd4194326418da33169a4f27346f8)
I20260808 18:29:04.830622 5813 raft_consensus.cc:482] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 181bd4194326418da33169a4f27346f8)
I20260808 18:29:04.830818 5813 raft_consensus.cc:504] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: false } }
I20260808 18:29:04.830840 5812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: false } }
I20260808 18:29:04.831869 5812 leader_election.cc:288] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 181bd4194326418da33169a4f27346f8 (127.29.78.190:34937), a692e2279f124613a8b62dc5c5cda6ce (127.29.78.189:32899)
I20260808 18:29:04.832243 5813 leader_election.cc:288] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 181bd4194326418da33169a4f27346f8 (127.29.78.190:34937), 12f6a1d683a94e278adf262ca5911ca6 (127.29.78.188:43147)
I20260808 18:29:04.832669 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.832104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.0.0.1:49448)
0808 18:29:04.832469 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:04.832551 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":57,"threads_started":1}
W20260808 18:29:04.832841 5230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
I20260808 18:29:04.832888 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.832079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.0.0.1:49446)
0808 18:29:04.832728 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:04.832803 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":94,"threads_started":1}
W20260808 18:29:04.833143 5023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
I20260808 18:29:04.833801 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.833559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.0.0.1:49468)
0808 18:29:04.833691 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:04.833738 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260808 18:29:04.833930 5230 leader_election.cc:334] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 181bd4194326418da33169a4f27346f8 (127.29.78.190:34937): Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
I20260808 18:29:04.834287 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.833904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.0.0.1:49480)
0808 18:29:04.834133 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:04.834221 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
W20260808 18:29:04.834441 5023 leader_election.cc:334] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 181bd4194326418da33169a4f27346f8 (127.29.78.190:34937): Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
I20260808 18:29:04.836597 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.832016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43147 (local address 127.0.0.1:50336)
0808 18:29:04.832388 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:04.832403 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:04.832515 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:04.833342 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:04.833346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:04.833365 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:04.833482 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:04.833493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:04.834553 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:04.834557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:04.835423 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:04.835432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:04.836123 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:04.836127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:04.836286 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:04.836307 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 18:29:04.836396 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 18:29:04.836470 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":44,"thread_start_us":71,"threads_started":1}
I20260808 18:29:04.836608 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.832415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50336 (local address 127.29.78.188:43147)
0808 18:29:04.833101 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:04.833107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:04.833123 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:29:04.833164 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:04.833167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:04.833209 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:04.833609 (+ 400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:04.833708 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:04.834387 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:04.835561 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:04.835949 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:04.836106 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:04.836166 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:04.836414 (+ 248us) server_negotiation.cc:299] Negotiation successful
0808 18:29:04.836484 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":53,"threads_started":1}
I20260808 18:29:04.837003 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.832207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:32899 (local address 127.0.0.1:54056)
0808 18:29:04.832555 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:04.832584 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:04.832690 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:04.833275 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:04.833278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:04.833298 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:04.833434 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:04.833443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:04.834625 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:04.834629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:04.835596 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:04.835606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:04.836528 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:04.836532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:04.836641 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:04.836690 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 18:29:04.836784 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 18:29:04.836871 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":69,"threads_started":1}
I20260808 18:29:04.837103 5819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.832642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54056 (local address 127.29.78.189:32899)
0808 18:29:04.832926 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:04.832934 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:04.832950 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:29:04.832998 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:04.833001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:04.833050 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:04.833135 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:04.833587 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:04.834478 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:04.835740 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:04.836394 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:04.836524 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:04.836581 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:04.836907 (+ 326us) server_negotiation.cc:299] Negotiation successful
0808 18:29:04.836980 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":210,"thread_start_us":82,"threads_started":1}
I20260808 18:29:04.837067 5257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a692e2279f124613a8b62dc5c5cda6ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "12f6a1d683a94e278adf262ca5911ca6" is_pre_election: true
I20260808 18:29:04.837296 5257 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a692e2279f124613a8b62dc5c5cda6ce in term 1.
I20260808 18:29:04.837659 5022 leader_election.cc:302] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12f6a1d683a94e278adf262ca5911ca6, a692e2279f124613a8b62dc5c5cda6ce; no voters: 181bd4194326418da33169a4f27346f8
I20260808 18:29:04.837918 5813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 18:29:04.838011 5813 raft_consensus.cc:482] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 1 FOLLOWER]: Starting leader election (detected failure of leader 181bd4194326418da33169a4f27346f8)
I20260808 18:29:04.837927 5049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12f6a1d683a94e278adf262ca5911ca6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a692e2279f124613a8b62dc5c5cda6ce" is_pre_election: true
I20260808 18:29:04.838071 5813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 1 FOLLOWER]: Advancing to term 2
I20260808 18:29:04.839289 5813 raft_consensus.cc:504] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: false } }
I20260808 18:29:04.839562 5049 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 12f6a1d683a94e278adf262ca5911ca6 in current term 2: Already voted for candidate a692e2279f124613a8b62dc5c5cda6ce in this term.
I20260808 18:29:04.839740 5813 leader_election.cc:288] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [CANDIDATE]: Term 2 election: Requested vote from peers 181bd4194326418da33169a4f27346f8 (127.29.78.190:34937), 12f6a1d683a94e278adf262ca5911ca6 (127.29.78.188:43147)
I20260808 18:29:04.840101 5257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a692e2279f124613a8b62dc5c5cda6ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "12f6a1d683a94e278adf262ca5911ca6"
I20260808 18:29:04.840243 5257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [term 1 FOLLOWER]: Advancing to term 2
I20260808 18:29:04.840328 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.840028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.0.0.1:49488)
0808 18:29:04.840208 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:04.840263 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260808 18:29:04.840320 5229 leader_election.cc:302] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 12f6a1d683a94e278adf262ca5911ca6; no voters: 181bd4194326418da33169a4f27346f8, a692e2279f124613a8b62dc5c5cda6ce
I20260808 18:29:04.841004 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.840730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.0.0.1:49504)
0808 18:29:04.840883 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:04.840938 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260808 18:29:04.841148 5023 leader_election.cc:334] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 181bd4194326418da33169a4f27346f8 (127.29.78.190:34937): Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
I20260808 18:29:04.841588 5257 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a692e2279f124613a8b62dc5c5cda6ce in term 2.
I20260808 18:29:04.841747 5812 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 18:29:04.841913 5022 leader_election.cc:302] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12f6a1d683a94e278adf262ca5911ca6, a692e2279f124613a8b62dc5c5cda6ce; no voters: 181bd4194326418da33169a4f27346f8
I20260808 18:29:04.842094 5813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 2 FOLLOWER]: Leader election won for term 2
I20260808 18:29:04.842331 5813 raft_consensus.cc:686] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [term 2 LEADER]: Becoming Leader. State: Replica: a692e2279f124613a8b62dc5c5cda6ce, State: Running, Role: LEADER
I20260808 18:29:04.842618 5813 consensus_queue.cc:237] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: false } }
I20260808 18:29:04.843675 5822 sys_catalog.cc:455] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader a692e2279f124613a8b62dc5c5cda6ce. Latest consensus state: current_term: 2 leader_uuid: "a692e2279f124613a8b62dc5c5cda6ce" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: false } } }
I20260808 18:29:04.843816 5822 sys_catalog.cc:458] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: This master's current role is: LEADER
I20260808 18:29:04.844183 5825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:29:04.846517 5825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:29:04.846908 5825 catalog_manager.cc:1259] Loaded cluster ID: 79b2d9f0f7a04df6b2088bd4ca8342eb
I20260808 18:29:04.846978 5825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:29:04.847535 5825 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:29:04.847913 5825 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce: Loaded TSK: 0
I20260808 18:29:04.848560 5825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:29:04.875197 5032 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:29:04.878203 5032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:29:04.881036 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.880781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.0.0.1:49530)
0808 18:29:04.880899 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:04.880961 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 18:29:04.881139 5257 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [term 2 FOLLOWER]: Refusing update from remote peer a692e2279f124613a8b62dc5c5cda6ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260808 18:29:04.881661 5822 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 18:29:04.881791 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.881557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.0.0.1:49538)
0808 18:29:04.881665 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:04.881718 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 18:29:04.881968 5023 consensus_peers.cc:487] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce -> Peer 181bd4194326418da33169a4f27346f8 (127.29.78.190:34937): Couldn't send request to peer 181bd4194326418da33169a4f27346f8. Status: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 18:29:04.883148 5812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a692e2279f124613a8b62dc5c5cda6ce. Latest consensus state: current_term: 2 leader_uuid: "a692e2279f124613a8b62dc5c5cda6ce" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: false } } }
I20260808 18:29:04.883275 5812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:29:04.884330 5813 sys_catalog.cc:455] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a692e2279f124613a8b62dc5c5cda6ce" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: false } } }
I20260808 18:29:04.884470 5813 sys_catalog.cc:458] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce [sys.catalog]: This master's current role is: LEADER
I20260808 18:29:04.884814 5812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a692e2279f124613a8b62dc5c5cda6ce" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "181bd4194326418da33169a4f27346f8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34937 } } peers { permanent_uuid: "a692e2279f124613a8b62dc5c5cda6ce" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 32899 } attrs { promote: false } } peers { permanent_uuid: "12f6a1d683a94e278adf262ca5911ca6" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43147 } attrs { promote: false } } }
I20260808 18:29:04.884931 5812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f6a1d683a94e278adf262ca5911ca6 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:29:04.895479 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.891498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60080 (local address 127.29.78.130:38341)
0808 18:29:04.892005 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:04.892011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:04.892036 (+ 25us) server_negotiation.cc:407] Connection header received
0808 18:29:04.892086 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:04.892089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:04.892195 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:04.892283 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:04.892544 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:04.893209 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:04.894196 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:04.894851 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:04.894952 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:04.895013 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:04.895266 (+ 253us) server_negotiation.cc:299] Negotiation successful
0808 18:29:04.895330 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":100,"threads_started":1}
I20260808 18:29:04.895483 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.891222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:38341 (local address 127.0.0.1:60080)
0808 18:29:04.891471 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:04.891488 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:04.891572 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:04.892317 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:04.892320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:04.892339 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:04.892421 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:04.892426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:04.893314 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:04.893318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:04.894072 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:04.894080 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:04.895070 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:04.895073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:04.895137 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:04.895155 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 18:29:04.895249 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 18:29:04.895319 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":87,"threads_started":1}
I20260808 18:29:04.895529 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.890370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:41395 (local address 127.0.0.1:55868)
0808 18:29:04.890814 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:04.890830 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:04.890924 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:04.891235 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:04.891238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:04.891255 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:04.891367 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:04.891375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:04.892598 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:04.892604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:04.893782 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:04.893792 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:04.895060 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:04.895064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:04.895171 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:04.895209 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 18:29:04.895284 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 18:29:04.895363 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346}
I20260808 18:29:04.895983 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.890550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55868 (local address 127.29.78.131:41395)
0808 18:29:04.890879 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:04.890884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:04.890901 (+ 17us) server_negotiation.cc:407] Connection header received
0808 18:29:04.891025 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:04.891031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:04.891089 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:04.891200 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:04.891526 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:04.892463 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:04.893945 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:04.894798 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:04.894921 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:04.894994 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:04.895745 (+ 751us) server_negotiation.cc:299] Negotiation successful
0808 18:29:04.895845 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":91,"threads_started":1}
I20260808 18:29:04.896072 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.890662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:46615 (local address 127.0.0.1:40698)
0808 18:29:04.890892 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:04.890906 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:04.890993 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:04.891805 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:04.891808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:04.891823 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:04.891919 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:04.891926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:04.893026 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:04.893030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:04.894239 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:04.894250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:04.895272 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:04.895279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:04.895381 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:04.895408 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 18:29:04.895461 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 18:29:04.895523 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260808 18:29:04.896147 5518 tablet_service.cc:1467] Processing CreateTablet for tablet 5384f82dbb214eba88203da61430f78b (DEFAULT_TABLE table=test-table [id=ef7d5168eed84e76af09ba4a5e07480e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:29:04.896472 5518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5384f82dbb214eba88203da61430f78b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:04.896651 5666 tablet_service.cc:1467] Processing CreateTablet for tablet 5384f82dbb214eba88203da61430f78b (DEFAULT_TABLE table=test-table [id=ef7d5168eed84e76af09ba4a5e07480e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:29:04.896879 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.890921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40698 (local address 127.29.78.129:46615)
0808 18:29:04.891539 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:04.891544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:04.891560 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:29:04.891618 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:04.891622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:04.891677 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:04.892035 (+ 358us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:04.892146 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:04.892901 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:04.894431 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:04.895127 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:04.896503 (+ 1376us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:04.896604 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:04.896644 (+ 40us) server_negotiation.cc:299] Negotiation successful
0808 18:29:04.896715 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":91,"threads_started":1}
I20260808 18:29:04.896919 5666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5384f82dbb214eba88203da61430f78b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:04.897401 5364 tablet_service.cc:1467] Processing CreateTablet for tablet 5384f82dbb214eba88203da61430f78b (DEFAULT_TABLE table=test-table [id=ef7d5168eed84e76af09ba4a5e07480e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:29:04.897737 5364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5384f82dbb214eba88203da61430f78b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:04.899489 5837 tablet_bootstrap.cc:492] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de: Bootstrap starting.
I20260808 18:29:04.899993 5836 tablet_bootstrap.cc:492] T 5384f82dbb214eba88203da61430f78b P 51548778d14f48d49953bdb29879b2ca: Bootstrap starting.
I20260808 18:29:04.900488 5837 tablet_bootstrap.cc:654] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:04.900810 5838 tablet_bootstrap.cc:492] T 5384f82dbb214eba88203da61430f78b P 01781749e0d0467da710bf3c66b32f88: Bootstrap starting.
I20260808 18:29:04.900997 5836 tablet_bootstrap.cc:654] T 5384f82dbb214eba88203da61430f78b P 51548778d14f48d49953bdb29879b2ca: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:04.901893 5837 tablet_bootstrap.cc:492] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de: No bootstrap required, opened a new log
I20260808 18:29:04.902002 5837 ts_tablet_manager.cc:1397] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:29:04.902244 5836 tablet_bootstrap.cc:492] T 5384f82dbb214eba88203da61430f78b P 51548778d14f48d49953bdb29879b2ca: No bootstrap required, opened a new log
I20260808 18:29:04.902345 5836 ts_tablet_manager.cc:1397] T 5384f82dbb214eba88203da61430f78b P 51548778d14f48d49953bdb29879b2ca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 18:29:04.902532 5837 raft_consensus.cc:348] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41395 } } peers { permanent_uuid: "01781749e0d0467da710bf3c66b32f88" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46615 } } peers { permanent_uuid: "51548778d14f48d49953bdb29879b2ca" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38341 } }
I20260808 18:29:04.902680 5837 raft_consensus.cc:374] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:04.902737 5837 raft_consensus.cc:729] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d47d6d7c4d445dbaebedcaae667f1de, State: Initialized, Role: FOLLOWER
I20260808 18:29:04.902784 5838 tablet_bootstrap.cc:654] T 5384f82dbb214eba88203da61430f78b P 01781749e0d0467da710bf3c66b32f88: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:04.902774 5836 raft_consensus.cc:348] T 5384f82dbb214eba88203da61430f78b P 51548778d14f48d49953bdb29879b2ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41395 } } peers { permanent_uuid: "01781749e0d0467da710bf3c66b32f88" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46615 } } peers { permanent_uuid: "51548778d14f48d49953bdb29879b2ca" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38341 } }
I20260808 18:29:04.902899 5836 raft_consensus.cc:374] T 5384f82dbb214eba88203da61430f78b P 51548778d14f48d49953bdb29879b2ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:04.902943 5836 raft_consensus.cc:729] T 5384f82dbb214eba88203da61430f78b P 51548778d14f48d49953bdb29879b2ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51548778d14f48d49953bdb29879b2ca, State: Initialized, Role: FOLLOWER
I20260808 18:29:04.902907 5837 consensus_queue.cc:260] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41395 } } peers { permanent_uuid: "01781749e0d0467da710bf3c66b32f88" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46615 } } peers { permanent_uuid: "51548778d14f48d49953bdb29879b2ca" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38341 } }
I20260808 18:29:04.903116 5836 consensus_queue.cc:260] T 5384f82dbb214eba88203da61430f78b P 51548778d14f48d49953bdb29879b2ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41395 } } peers { permanent_uuid: "01781749e0d0467da710bf3c66b32f88" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46615 } } peers { permanent_uuid: "51548778d14f48d49953bdb29879b2ca" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38341 } }
I20260808 18:29:04.903328 5837 ts_tablet_manager.cc:1428] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:29:04.903857 5836 ts_tablet_manager.cc:1428] T 5384f82dbb214eba88203da61430f78b P 51548778d14f48d49953bdb29879b2ca: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260808 18:29:04.904183 5844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.903766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.29.78.131:57735)
0808 18:29:04.904022 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:04.904089 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":119,"threads_started":1}
I20260808 18:29:04.905009 5845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.904607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.29.78.130:37453)
0808 18:29:04.904857 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:04.904919 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":107,"threads_started":1}
I20260808 18:29:04.905617 5845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.905409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.29.78.130:48969)
0808 18:29:04.905516 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:04.905562 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 18:29:04.906055 5739 tablet.cc:2369] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:29:04.906888 5738 heartbeater.cc:503] Master 127.29.78.189:32899 was elected leader, sending a full tablet report...
I20260808 18:29:04.907689 5844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.907405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.29.78.131:51313)
0808 18:29:04.907560 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:04.907614 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260808 18:29:04.908887 5590 heartbeater.cc:503] Master 127.29.78.189:32899 was elected leader, sending a full tablet report...
I20260808 18:29:04.909035 5838 tablet_bootstrap.cc:492] T 5384f82dbb214eba88203da61430f78b P 01781749e0d0467da710bf3c66b32f88: No bootstrap required, opened a new log
I20260808 18:29:04.909164 5838 ts_tablet_manager.cc:1397] T 5384f82dbb214eba88203da61430f78b P 01781749e0d0467da710bf3c66b32f88: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260808 18:29:04.909710 5838 raft_consensus.cc:348] T 5384f82dbb214eba88203da61430f78b P 01781749e0d0467da710bf3c66b32f88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41395 } } peers { permanent_uuid: "01781749e0d0467da710bf3c66b32f88" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46615 } } peers { permanent_uuid: "51548778d14f48d49953bdb29879b2ca" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38341 } }
I20260808 18:29:04.909868 5838 raft_consensus.cc:374] T 5384f82dbb214eba88203da61430f78b P 01781749e0d0467da710bf3c66b32f88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:04.909930 5838 raft_consensus.cc:729] T 5384f82dbb214eba88203da61430f78b P 01781749e0d0467da710bf3c66b32f88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01781749e0d0467da710bf3c66b32f88, State: Initialized, Role: FOLLOWER
I20260808 18:29:04.910075 5838 consensus_queue.cc:260] T 5384f82dbb214eba88203da61430f78b P 01781749e0d0467da710bf3c66b32f88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41395 } } peers { permanent_uuid: "01781749e0d0467da710bf3c66b32f88" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46615 } } peers { permanent_uuid: "51548778d14f48d49953bdb29879b2ca" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38341 } }
I20260808 18:29:04.910495 5838 ts_tablet_manager.cc:1428] T 5384f82dbb214eba88203da61430f78b P 01781749e0d0467da710bf3c66b32f88: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260808 18:29:04.911257 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.910813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.29.78.129:58063)
0808 18:29:04.911096 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:04.911154 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":109,"threads_started":1}
I20260808 18:29:04.912047 5848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:04.911669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.29.78.129:34467)
0808 18:29:04.911889 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:04.911942 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":106,"threads_started":1}
I20260808 18:29:04.912267 5438 heartbeater.cc:503] Master 127.29.78.189:32899 was elected leader, sending a full tablet report...
W20260808 18:29:04.946211 5591 tablet.cc:2369] T 5384f82dbb214eba88203da61430f78b P 51548778d14f48d49953bdb29879b2ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:29:04.997608 5841 raft_consensus.cc:482] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:29:04.997756 5841 raft_consensus.cc:504] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41395 } } peers { permanent_uuid: "01781749e0d0467da710bf3c66b32f88" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46615 } } peers { permanent_uuid: "51548778d14f48d49953bdb29879b2ca" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38341 } }
I20260808 18:29:04.998266 5841 leader_election.cc:288] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01781749e0d0467da710bf3c66b32f88 (127.29.78.129:46615), 51548778d14f48d49953bdb29879b2ca (127.29.78.130:38341)
I20260808 18:29:04.998754 5539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5384f82dbb214eba88203da61430f78b" candidate_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51548778d14f48d49953bdb29879b2ca" is_pre_election: true
I20260808 18:29:04.998755 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5384f82dbb214eba88203da61430f78b" candidate_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01781749e0d0467da710bf3c66b32f88" is_pre_election: true
I20260808 18:29:04.998925 5539 raft_consensus.cc:2454] T 5384f82dbb214eba88203da61430f78b P 51548778d14f48d49953bdb29879b2ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d47d6d7c4d445dbaebedcaae667f1de in term 0.
I20260808 18:29:04.998955 5386 raft_consensus.cc:2454] T 5384f82dbb214eba88203da61430f78b P 01781749e0d0467da710bf3c66b32f88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d47d6d7c4d445dbaebedcaae667f1de in term 0.
I20260808 18:29:04.999318 5615 leader_election.cc:302] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01781749e0d0467da710bf3c66b32f88, 5d47d6d7c4d445dbaebedcaae667f1de; no voters:
I20260808 18:29:04.999502 5841 raft_consensus.cc:2781] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:29:04.999565 5841 raft_consensus.cc:482] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:29:04.999593 5841 raft_consensus.cc:3037] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:05.000705 5841 raft_consensus.cc:504] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41395 } } peers { permanent_uuid: "01781749e0d0467da710bf3c66b32f88" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46615 } } peers { permanent_uuid: "51548778d14f48d49953bdb29879b2ca" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38341 } }
I20260808 18:29:05.001117 5841 leader_election.cc:288] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de [CANDIDATE]: Term 1 election: Requested vote from peers 01781749e0d0467da710bf3c66b32f88 (127.29.78.129:46615), 51548778d14f48d49953bdb29879b2ca (127.29.78.130:38341)
I20260808 18:29:05.001422 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5384f82dbb214eba88203da61430f78b" candidate_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01781749e0d0467da710bf3c66b32f88"
I20260808 18:29:05.001509 5539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5384f82dbb214eba88203da61430f78b" candidate_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51548778d14f48d49953bdb29879b2ca"
I20260808 18:29:05.001626 5386 raft_consensus.cc:3037] T 5384f82dbb214eba88203da61430f78b P 01781749e0d0467da710bf3c66b32f88 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:05.001647 5539 raft_consensus.cc:3037] T 5384f82dbb214eba88203da61430f78b P 51548778d14f48d49953bdb29879b2ca [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:05.002727 5539 raft_consensus.cc:2454] T 5384f82dbb214eba88203da61430f78b P 51548778d14f48d49953bdb29879b2ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d47d6d7c4d445dbaebedcaae667f1de in term 1.
I20260808 18:29:05.002729 5386 raft_consensus.cc:2454] T 5384f82dbb214eba88203da61430f78b P 01781749e0d0467da710bf3c66b32f88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d47d6d7c4d445dbaebedcaae667f1de in term 1.
I20260808 18:29:05.003036 5613 leader_election.cc:302] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51548778d14f48d49953bdb29879b2ca, 5d47d6d7c4d445dbaebedcaae667f1de; no voters:
I20260808 18:29:05.003212 5841 raft_consensus.cc:2781] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:29:05.003440 5841 raft_consensus.cc:686] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de [term 1 LEADER]: Becoming Leader. State: Replica: 5d47d6d7c4d445dbaebedcaae667f1de, State: Running, Role: LEADER
I20260808 18:29:05.003602 5841 consensus_queue.cc:237] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41395 } } peers { permanent_uuid: "01781749e0d0467da710bf3c66b32f88" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46615 } } peers { permanent_uuid: "51548778d14f48d49953bdb29879b2ca" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38341 } }
I20260808 18:29:05.004427 5844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:05.004142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.29.78.131:49631)
0808 18:29:05.004281 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:05.004351 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 18:29:05.005425 5032 catalog_manager.cc:5495] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d47d6d7c4d445dbaebedcaae667f1de (127.29.78.131). New cstate: current_term: 1 leader_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d47d6d7c4d445dbaebedcaae667f1de" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01781749e0d0467da710bf3c66b32f88" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51548778d14f48d49953bdb29879b2ca" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38341 } health_report { overall_health: UNKNOWN } } }
I20260808 18:29:05.017616 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:05.014443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55882 (local address 127.29.78.131:41395)
0808 18:29:05.014552 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:05.014557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:05.014601 (+ 44us) server_negotiation.cc:407] Connection header received
0808 18:29:05.014763 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:05.014766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:05.014817 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:05.014896 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:29:05.015166 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:05.015689 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:05.016547 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:05.016772 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:05.016876 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:05.017171 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:29:05.017239 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:29:05.017317 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:05.017408 (+ 91us) server_negotiation.cc:299] Negotiation successful
0808 18:29:05.017472 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 18:29:05.019807 5539 raft_consensus.cc:1261] T 5384f82dbb214eba88203da61430f78b P 51548778d14f48d49953bdb29879b2ca [term 1 FOLLOWER]: Refusing update from remote peer 5d47d6d7c4d445dbaebedcaae667f1de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:29:05.019954 5386 raft_consensus.cc:1261] T 5384f82dbb214eba88203da61430f78b P 01781749e0d0467da710bf3c66b32f88 [term 1 FOLLOWER]: Refusing update from remote peer 5d47d6d7c4d445dbaebedcaae667f1de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:29:05.020282 5841 consensus_queue.cc:1035] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "51548778d14f48d49953bdb29879b2ca" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:29:05.020452 5849 consensus_queue.cc:1035] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "01781749e0d0467da710bf3c66b32f88" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:29:05.022001 5848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:05.021699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.29.78.129:49817)
0808 18:29:05.021819 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:05.021882 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 18:29:05.024067 5845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:05.023810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.29.78.130:40923)
0808 18:29:05.023919 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:05.023980 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 18:29:05.025934 5844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:05.025666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.29.78.131:45943)
0808 18:29:05.025786 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:05.025844 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 18:29:05.026305 5848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:05.026083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.29.78.129:37303)
0808 18:29:05.026176 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:05.026235 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260808 18:29:05.028152 5845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:05.027758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34937 (local address 127.29.78.130:32971)
0808 18:29:05.028012 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:05.028084 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260808 18:29:05.032949 5032 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:29:05.033082 5032 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:29:05.036675 5032 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a692e2279f124613a8b62dc5c5cda6ce: Sending DeleteTablet for 3 replicas of tablet 5384f82dbb214eba88203da61430f78b
I20260808 18:29:05.037400 5518 tablet_service.cc:1514] Processing DeleteTablet for tablet 5384f82dbb214eba88203da61430f78b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:29:05 UTC) from {username='oryx', principal='oryx/127.29.78.189@KRBTEST.COM'} at 127.0.0.1:60080
I20260808 18:29:05.037539 5364 tablet_service.cc:1514] Processing DeleteTablet for tablet 5384f82dbb214eba88203da61430f78b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:29:05 UTC) from {username='oryx', principal='oryx/127.29.78.189@KRBTEST.COM'} at 127.0.0.1:40698
I20260808 18:29:05.037585 5666 tablet_service.cc:1514] Processing DeleteTablet for tablet 5384f82dbb214eba88203da61430f78b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:29:05 UTC) from {username='oryx', principal='oryx/127.29.78.189@KRBTEST.COM'} at 127.0.0.1:55868
I20260808 18:29:05.038048 5861 tablet_replica.cc:330] T 5384f82dbb214eba88203da61430f78b P 51548778d14f48d49953bdb29879b2ca: stopping tablet replica
I20260808 18:29:05.038113 5862 tablet_replica.cc:330] T 5384f82dbb214eba88203da61430f78b P 01781749e0d0467da710bf3c66b32f88: stopping tablet replica
I20260808 18:29:05.038172 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 5204
I20260808 18:29:05.038300 5861 raft_consensus.cc:2229] T 5384f82dbb214eba88203da61430f78b P 51548778d14f48d49953bdb29879b2ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:29:05.038482 5861 raft_consensus.cc:2258] T 5384f82dbb214eba88203da61430f78b P 51548778d14f48d49953bdb29879b2ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:29:05.038897 5863 tablet_replica.cc:330] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de: stopping tablet replica
I20260808 18:29:05.039047 5863 raft_consensus.cc:2229] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:29:05.039278 5863 raft_consensus.cc:2258] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:29:05.039388 5861 ts_tablet_manager.cc:1905] T 5384f82dbb214eba88203da61430f78b P 51548778d14f48d49953bdb29879b2ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:29:05.043164 5861 ts_tablet_manager.cc:1918] T 5384f82dbb214eba88203da61430f78b P 51548778d14f48d49953bdb29879b2ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:29:05.043336 5861 log.cc:1196] T 5384f82dbb214eba88203da61430f78b P 51548778d14f48d49953bdb29879b2ca: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/5384f82dbb214eba88203da61430f78b
I20260808 18:29:05.043656 5863 ts_tablet_manager.cc:1905] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:29:05.043792 5861 ts_tablet_manager.cc:1939] T 5384f82dbb214eba88203da61430f78b P 51548778d14f48d49953bdb29879b2ca: Deleting consensus metadata
I20260808 18:29:05.044639 5022 catalog_manager.cc:4841] TS 51548778d14f48d49953bdb29879b2ca (127.29.78.130:38341): tablet 5384f82dbb214eba88203da61430f78b (table test-table [id=ef7d5168eed84e76af09ba4a5e07480e]) successfully deleted
I20260808 18:29:05.047021 5863 ts_tablet_manager.cc:1918] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:29:05.047125 5863 log.cc:1196] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/5384f82dbb214eba88203da61430f78b
I20260808 18:29:05.047448 5863 ts_tablet_manager.cc:1939] T 5384f82dbb214eba88203da61430f78b P 5d47d6d7c4d445dbaebedcaae667f1de: Deleting consensus metadata
I20260808 18:29:05.048242 5023 catalog_manager.cc:4841] TS 5d47d6d7c4d445dbaebedcaae667f1de (127.29.78.131:41395): tablet 5384f82dbb214eba88203da61430f78b (table test-table [id=ef7d5168eed84e76af09ba4a5e07480e]) successfully deleted
I20260808 18:29:05.050549 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 5445
I20260808 18:29:05.052141 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:05.050641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:46615 (local address 127.0.0.1:40708)
0808 18:29:05.051949 (+ 1308us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:05.052028 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1198}
W20260808 18:29:05.054579 5024 catalog_manager.cc:4584] TS 01781749e0d0467da710bf3c66b32f88 (127.29.78.129:46615): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5384f82dbb214eba88203da61430f78b: Network error: Client connection negotiation failed: client connection to 127.29.78.129:46615: connect: Connection refused (error 111)
I20260808 18:29:05.059674 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 5595
I20260808 18:29:05.069368 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 4884
I20260808 18:29:05.076499 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 5001
2026-08-08T18:29:05Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5439 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:29:05 dist-test-slave-gp6n krb5kdc[5866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:29:05 dist-test-slave-gp6n krb5kdc[5866](info): set up 2 sockets
Aug 08 18:29:05 dist-test-slave-gp6n krb5kdc[5866](info): commencing operation
krb5kdc: starting...
W20260808 18:29:06.185456 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:29:06 dist-test-slave-gp6n krb5kdc[5866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:29:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:29:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.190.keytab.
Entry for principal oryx/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.190.keytab.
I20260808 18:29:06.326546 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:42985
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.190.keytab
--principal=oryx/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:42985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:29:06.440397 5880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:06.440644 5880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:06.440683 5880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:06.445063 5880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:29:06.445137 5880 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 18:29:06.445158 5880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:06.445178 5880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:06.445197 5880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:29:06.445277 5880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:29:06.451005 5880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:42985
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.190.keytab
--principal=oryx/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:42985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:06.452009 5880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:06.453039 5880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:06.461633 5890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:06.461724 5880 server_base.cc:1034] running on GCE node
W20260808 18:29:06.461655 5888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:06.461653 5886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:06.462512 5880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:06.463729 5880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:06.464946 5880 hybrid_clock.cc:648] HybridClock initialized: now 1786213746464919 us; error 39 us; skew 500 ppm
Aug 08 18:29:06 dist-test-slave-gp6n krb5kdc[5866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213746, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:06.467803 5880 init.cc:377] Logged in from keytab as oryx/127.29.78.190@KRBTEST.COM (short username oryx)
I20260808 18:29:06.468744 5880 webserver.cc:460] Webserver started at http://127.29.78.190:39625/ using document root <none> and password file <none>
I20260808 18:29:06.469298 5880 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:06.469408 5880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:06.469642 5880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:06.471755 5880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "c651e634ac5249109a0abd50e8d32fad"
format_stamp: "Formatted at 2026-08-08 18:29:06 on dist-test-slave-gp6n"
server_key: "53c94b8b6f4bb57270eb095d9587010c"
server_key_iv: "6857a4a931c70cac25f5b84018c70153"
server_key_version: "encryptionkey@0"
I20260808 18:29:06.472321 5880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "c651e634ac5249109a0abd50e8d32fad"
format_stamp: "Formatted at 2026-08-08 18:29:06 on dist-test-slave-gp6n"
server_key: "53c94b8b6f4bb57270eb095d9587010c"
server_key_iv: "6857a4a931c70cac25f5b84018c70153"
server_key_version: "encryptionkey@0"
I20260808 18:29:06.476104 5880 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:29:06.478600 5898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:06.479928 5880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:29:06.480088 5880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "c651e634ac5249109a0abd50e8d32fad"
format_stamp: "Formatted at 2026-08-08 18:29:06 on dist-test-slave-gp6n"
server_key: "53c94b8b6f4bb57270eb095d9587010c"
server_key_iv: "6857a4a931c70cac25f5b84018c70153"
server_key_version: "encryptionkey@0"
I20260808 18:29:06.480208 5880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:06.493235 5880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:06.496570 5880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:06.496804 5880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:06.506000 5880 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:42985
I20260808 18:29:06.506026 5962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:42985 every 8 connection(s)
I20260808 18:29:06.507174 5880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:29:06.510084 5963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:06.513428 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 5880
I20260808 18:29:06.513576 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:29:06.513906 30010 external_mini_cluster.cc:1428] Setting key 79e361a145619f585ac12377bfad2b26
I20260808 18:29:06.516405 5963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad: Bootstrap starting.
I20260808 18:29:06.518716 5963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:06.519484 5963 log.cc:824] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad: Log is configured to *not* fsync() on all Append() calls
I20260808 18:29:06.521355 5963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad: No bootstrap required, opened a new log
Aug 08 18:29:06 dist-test-slave-gp6n krb5kdc[5866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260808 18:29:06.525722 5963 raft_consensus.cc:348] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c651e634ac5249109a0abd50e8d32fad" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42985 } }
I20260808 18:29:06.526062 5963 raft_consensus.cc:374] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:06.526127 5963 raft_consensus.cc:729] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c651e634ac5249109a0abd50e8d32fad, State: Initialized, Role: FOLLOWER
I20260808 18:29:06.526645 5963 consensus_queue.cc:260] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c651e634ac5249109a0abd50e8d32fad" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42985 } }
I20260808 18:29:06.526816 5963 raft_consensus.cc:388] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:29:06.526885 5963 raft_consensus.cc:482] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:29:06.527017 5963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:06.528247 5963 raft_consensus.cc:504] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c651e634ac5249109a0abd50e8d32fad" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42985 } }
I20260808 18:29:06.528604 5963 leader_election.cc:302] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c651e634ac5249109a0abd50e8d32fad; no voters:
I20260808 18:29:06.528966 5963 leader_election.cc:288] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:29:06.529147 5969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:29:06.529480 5969 raft_consensus.cc:686] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad [term 1 LEADER]: Becoming Leader. State: Replica: c651e634ac5249109a0abd50e8d32fad, State: Running, Role: LEADER
I20260808 18:29:06.529824 5969 consensus_queue.cc:237] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c651e634ac5249109a0abd50e8d32fad" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42985 } }
I20260808 18:29:06.530263 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:06.515426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41796 (local address 127.29.78.190:42985)
0808 18:29:06.515875 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:06.515892 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:06.515938 (+ 46us) server_negotiation.cc:407] Connection header received
0808 18:29:06.516631 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:06.516650 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:06.516993 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:06.517332 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:06.518233 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:06.519820 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:06.520822 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:06.521448 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:06.521648 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:06.524610 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:06.524616 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:06.524633 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:06.524665 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:06.526781 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:06.527445 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:06.527451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:06.527457 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:06.527592 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:06.527880 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:06.527885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:06.527889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:06.528435 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:06.528640 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:06.528978 (+ 338us) server_negotiation.cc:299] Negotiation successful
0808 18:29:06.529297 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":139,"threads_started":1}
I20260808 18:29:06.530313 5963 sys_catalog.cc:564] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:29:06.531750 5970 sys_catalog.cc:455] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c651e634ac5249109a0abd50e8d32fad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c651e634ac5249109a0abd50e8d32fad" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42985 } } }
I20260808 18:29:06.531877 5970 sys_catalog.cc:458] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad [sys.catalog]: This master's current role is: LEADER
I20260808 18:29:06.531725 5971 sys_catalog.cc:455] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad [sys.catalog]: SysCatalogTable state changed. Reason: New leader c651e634ac5249109a0abd50e8d32fad. Latest consensus state: current_term: 1 leader_uuid: "c651e634ac5249109a0abd50e8d32fad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c651e634ac5249109a0abd50e8d32fad" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42985 } } }
I20260808 18:29:06.531978 5971 sys_catalog.cc:458] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad [sys.catalog]: This master's current role is: LEADER
I20260808 18:29:06.532493 5982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:29:06.536005 5982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:29:06.541432 5982 catalog_manager.cc:1347] Generated new cluster ID: 5de43245633b4ad1ae28db90af306bc6
I20260808 18:29:06.541497 5982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:29:06.557941 5982 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:29:06.558975 5982 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:29:06.565487 5982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad: Generated new TSK 0
I20260808 18:29:06.566172 5982 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.129.keytab.
Entry for principal oryx/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.129.keytab.
I20260808 18:29:06.602523 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:42985
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.129.keytab
--principal=oryx/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:29:06.713374 5991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:06.713654 5991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:06.713734 5991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:06.717742 5991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:06.717826 5991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:06.717897 5991 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 18:29:06.717978 5991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:29:06.723057 5991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.129.keytab
--principal=oryx/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.78.190:42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:06.724042 5991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:06.725067 5991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:06.734565 5999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:06.734622 5997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:06.734565 5996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:06.734767 5991 server_base.cc:1034] running on GCE node
I20260808 18:29:06.735212 5991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:06.735853 5991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:06.737046 5991 hybrid_clock.cc:648] HybridClock initialized: now 1786213746737033 us; error 26 us; skew 500 ppm
Aug 08 18:29:06 dist-test-slave-gp6n krb5kdc[5866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213746, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:06.744906 5991 init.cc:377] Logged in from keytab as oryx/127.29.78.129@KRBTEST.COM (short username oryx)
I20260808 18:29:06.745741 5991 webserver.cc:460] Webserver started at http://127.29.78.129:41165/ using document root <none> and password file <none>
I20260808 18:29:06.746258 5991 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:06.746316 5991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:06.746562 5991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:06.748580 5991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "b992bf381f0a41849ffc437f30c71bce"
format_stamp: "Formatted at 2026-08-08 18:29:06 on dist-test-slave-gp6n"
server_key: "106d52ab691e1f1cbfb2a68f8f0407c1"
server_key_iv: "3905b495857b7baa746aeadb63dc5897"
server_key_version: "encryptionkey@0"
I20260808 18:29:06.749156 5991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "b992bf381f0a41849ffc437f30c71bce"
format_stamp: "Formatted at 2026-08-08 18:29:06 on dist-test-slave-gp6n"
server_key: "106d52ab691e1f1cbfb2a68f8f0407c1"
server_key_iv: "3905b495857b7baa746aeadb63dc5897"
server_key_version: "encryptionkey@0"
I20260808 18:29:06.752882 5991 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 18:29:06.755285 6008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:06.756412 5991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:29:06.756554 5991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "b992bf381f0a41849ffc437f30c71bce"
format_stamp: "Formatted at 2026-08-08 18:29:06 on dist-test-slave-gp6n"
server_key: "106d52ab691e1f1cbfb2a68f8f0407c1"
server_key_iv: "3905b495857b7baa746aeadb63dc5897"
server_key_version: "encryptionkey@0"
I20260808 18:29:06.756666 5991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:06.774075 5991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:06.777245 5991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:06.777456 5991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:06.779089 5991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:06.779162 5991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:06.779237 5991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:06.779273 5991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 18:29:06 dist-test-slave-gp6n krb5kdc[5866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213746, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260808 18:29:06.790686 5991 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:39945
I20260808 18:29:06.790729 6131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:39945 every 8 connection(s)
I20260808 18:29:06.791702 5991 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:29:06.793524 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:06.780173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:55293 (local address 127.29.78.190:42985)
0808 18:29:06.780336 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:06.780341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:06.781220 (+ 879us) server_negotiation.cc:407] Connection header received
0808 18:29:06.782347 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:06.782350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:06.782429 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:06.782552 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:06.783901 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:06.784560 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:06.785794 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:06.786103 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:06.786225 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:06.789428 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:06.789434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:06.789440 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:06.789475 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:06.791223 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:06.791817 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:06.791820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:06.791822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:06.791926 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:06.792347 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:06.792351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:06.792353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:06.792608 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:06.792734 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:06.793222 (+ 488us) server_negotiation.cc:299] Negotiation successful
0808 18:29:06.793349 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 18:29:06.794255 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:06.780575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42985 (local address 127.29.78.129:55293)
0808 18:29:06.781043 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:06.781093 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:06.782125 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:06.782730 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:06.782737 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:06.783066 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:06.783737 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:06.783755 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:06.784726 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:06.784730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:06.785664 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:06.785674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:06.786285 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:06.786289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:06.786525 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:06.787203 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:06.787218 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:06.789211 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:06.791340 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:06.791348 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:06.791365 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:06.791704 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:06.792034 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:06.792039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:06.792042 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:06.792220 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:06.792745 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:06.792753 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:06.792986 (+ 233us) client_negotiation.cc:769] Sending connection context
0808 18:29:06.793175 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 18:29:06.793481 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":136,"threads_started":1}
I20260808 18:29:06.795492 6132 heartbeater.cc:347] Connected to a master server at 127.29.78.190:42985
I20260808 18:29:06.795626 6132 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:06.796159 6132 heartbeater.cc:511] Master 127.29.78.190:42985 requested a full tablet report, sending...
I20260808 18:29:06.797778 5914 ts_manager.cc:194] Registered new tserver with Master: b992bf381f0a41849ffc437f30c71bce (127.29.78.129:39945)
I20260808 18:29:06.798983 5914 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:55293
I20260808 18:29:06.799764 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 5991
I20260808 18:29:06.799901 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:29:06.800213 30010 external_mini_cluster.cc:1428] Setting key 3a4778814334353695988ca5a52e2deb
I20260808 18:29:06.805208 6140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:06.797692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42985 (local address 127.29.78.129:54361)
0808 18:29:06.797993 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:06.798006 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:06.798120 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:06.798335 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:06.798337 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:06.798565 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:06.798668 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:06.798673 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:06.799460 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:06.799464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:06.800309 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:06.800319 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:06.800836 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:06.800839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:06.800938 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:06.801449 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:06.801465 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:06.801924 (+ 459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:06.803800 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:06.803806 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:06.803809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:06.804018 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:06.804317 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:06.804319 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:06.804320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:06.804393 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:06.804819 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:06.804821 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:06.804890 (+ 69us) client_negotiation.cc:769] Sending connection context
0808 18:29:06.804967 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 18:29:06.805074 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":104,"threads_started":1}
I20260808 18:29:06.805418 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:06.797775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:54361 (local address 127.29.78.190:42985)
0808 18:29:06.797927 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:06.797931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:06.798080 (+ 149us) server_negotiation.cc:407] Connection header received
0808 18:29:06.798196 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:06.798199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:06.798241 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:06.798321 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:06.798779 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:06.799322 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:06.800434 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:06.800713 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:06.800833 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:06.802059 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:06.802062 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:06.802065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:06.802090 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:06.803668 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:06.804108 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:06.804114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:06.804117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:06.804219 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:06.804483 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:06.804488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:06.804490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:06.804726 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:06.804829 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:06.805065 (+ 236us) server_negotiation.cc:299] Negotiation successful
0808 18:29:06.805195 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":58}
WARNING: no policy specified for oryx/127.29.78.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.130.keytab.
Entry for principal oryx/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.130.keytab.
I20260808 18:29:06.830472 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:42985
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.130.keytab
--principal=oryx/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:29:06.944518 6143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:06.944753 6143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:06.944795 6143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:06.948729 6143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:06.948779 6143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:06.948817 6143 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 18:29:06.948855 6143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:29:06.953950 6143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.130.keytab
--principal=oryx/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.78.190:42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:06.954991 6143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:06.955942 6143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:29:06.965358 6143 server_base.cc:1034] running on GCE node
W20260808 18:29:06.965349 6148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:06.965332 6152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:06.965389 6150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:06.966046 6143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:06.966658 6143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:06.967819 6143 hybrid_clock.cc:648] HybridClock initialized: now 1786213746967801 us; error 56 us; skew 500 ppm
Aug 08 18:29:06 dist-test-slave-gp6n krb5kdc[5866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213746, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:06.970597 6143 init.cc:377] Logged in from keytab as oryx/127.29.78.130@KRBTEST.COM (short username oryx)
I20260808 18:29:06.971449 6143 webserver.cc:460] Webserver started at http://127.29.78.130:40235/ using document root <none> and password file <none>
I20260808 18:29:06.971963 6143 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:06.972020 6143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:06.972191 6143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:06.974164 6143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "1108a5faf487493bb7703e1cae80a1e2"
format_stamp: "Formatted at 2026-08-08 18:29:06 on dist-test-slave-gp6n"
server_key: "8ebb1d9d6b3755eaffce322c0fbc5c6c"
server_key_iv: "6695194eddc53baf8dae49a721137ffc"
server_key_version: "encryptionkey@0"
I20260808 18:29:06.974759 6143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "1108a5faf487493bb7703e1cae80a1e2"
format_stamp: "Formatted at 2026-08-08 18:29:06 on dist-test-slave-gp6n"
server_key: "8ebb1d9d6b3755eaffce322c0fbc5c6c"
server_key_iv: "6695194eddc53baf8dae49a721137ffc"
server_key_version: "encryptionkey@0"
I20260808 18:29:06.978437 6143 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:29:06.980788 6159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:06.982009 6143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:29:06.982113 6143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "1108a5faf487493bb7703e1cae80a1e2"
format_stamp: "Formatted at 2026-08-08 18:29:06 on dist-test-slave-gp6n"
server_key: "8ebb1d9d6b3755eaffce322c0fbc5c6c"
server_key_iv: "6695194eddc53baf8dae49a721137ffc"
server_key_version: "encryptionkey@0"
I20260808 18:29:06.982184 6143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:06.992515 6143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:06.995664 6143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:06.995900 6143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:06.997331 6143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:06.997377 6143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:06.997416 6143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:06.997430 6143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 18:29:07 dist-test-slave-gp6n krb5kdc[5866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213746, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260808 18:29:07.008302 6143 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:42051
I20260808 18:29:07.008492 6279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:42051 every 8 connection(s)
I20260808 18:29:07.009243 6143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:29:07.012542 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:06.998354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:44505 (local address 127.29.78.190:42985)
0808 18:29:06.998527 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:06.998532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:06.999520 (+ 988us) server_negotiation.cc:407] Connection header received
0808 18:29:07.001016 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.001020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.001085 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.001194 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:07.002851 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.003858 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.005183 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.005442 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.005567 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.008575 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:07.008581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:07.008583 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:07.008616 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:07.010378 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:07.010947 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:07.010953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:07.010955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:07.011094 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:07.011453 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:07.011457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:07.011458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:07.011650 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:07.011760 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:07.012238 (+ 478us) server_negotiation.cc:299] Negotiation successful
0808 18:29:07.012368 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 18:29:07.013290 6169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:06.998805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42985 (local address 127.29.78.130:44505)
0808 18:29:06.999343 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:06.999399 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:07.000759 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:07.001406 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:07.001415 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:07.001940 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:07.002704 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.002725 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.003996 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.004000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.005035 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.005045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.005606 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.005610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.005806 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.006513 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:07.006536 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:07.008390 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:07.010559 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:07.010564 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:07.010577 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:07.010829 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:07.011194 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:07.011197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:07.011199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:07.011345 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:07.011758 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:07.011765 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:07.011996 (+ 231us) client_negotiation.cc:769] Sending connection context
0808 18:29:07.012178 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 18:29:07.012487 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":154,"threads_started":1}
I20260808 18:29:07.014559 6280 heartbeater.cc:347] Connected to a master server at 127.29.78.190:42985
I20260808 18:29:07.014752 6280 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:07.015255 6280 heartbeater.cc:511] Master 127.29.78.190:42985 requested a full tablet report, sending...
I20260808 18:29:07.016297 5915 ts_manager.cc:194] Registered new tserver with Master: 1108a5faf487493bb7703e1cae80a1e2 (127.29.78.130:42051)
I20260808 18:29:07.016999 5915 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:44505
I20260808 18:29:07.018362 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 6143
I20260808 18:29:07.018537 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:29:07.018864 30010 external_mini_cluster.cc:1428] Setting key a49137b7411d7fc0d5e4180625967646
I20260808 18:29:07.024631 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.016818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:33841 (local address 127.29.78.190:42985)
0808 18:29:07.016941 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.016951 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.017172 (+ 221us) server_negotiation.cc:407] Connection header received
0808 18:29:07.017397 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.017400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.017444 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.017541 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:07.018047 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.018694 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.019686 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.019893 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.020017 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.021160 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:07.021164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:07.021166 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:07.021189 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:07.022945 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:07.023369 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:07.023375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:07.023376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:07.023475 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:07.023794 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:07.023797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:07.023798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:07.024009 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:07.024087 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:07.024430 (+ 343us) server_negotiation.cc:299] Negotiation successful
0808 18:29:07.024523 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 18:29:07.024804 6287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.016702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42985 (local address 127.29.78.130:33841)
0808 18:29:07.017077 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:07.017089 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:07.017259 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:07.017586 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:07.017589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:07.017827 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:07.017936 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.017943 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.018811 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.018815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.019571 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.019583 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.020051 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.020055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.020146 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.020695 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:07.020711 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:07.021026 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:07.023057 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:07.023060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:07.023061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:07.023255 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:07.023576 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:07.023578 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:07.023580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:07.023695 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:07.024110 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:07.024116 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:07.024230 (+ 114us) client_negotiation.cc:769] Sending connection context
0808 18:29:07.024347 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 18:29:07.024514 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for oryx/127.29.78.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.131.keytab.
Entry for principal oryx/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.131.keytab.
I20260808 18:29:07.050179 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:42985
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.131.keytab
--principal=oryx/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:44733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:29:07.164577 6290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:07.164875 6290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:07.164955 6290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:07.169013 6290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:07.169108 6290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:07.169178 6290 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 18:29:07.169250 6290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:29:07.174683 6290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.131.keytab
--principal=oryx/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.78.190:42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:07.175838 6290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:07.176921 6290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:07.186807 6297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:07.186828 6300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:07.186797 6298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:07.187235 6290 server_base.cc:1034] running on GCE node
I20260808 18:29:07.187649 6290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:07.188297 6290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:07.189483 6290 hybrid_clock.cc:648] HybridClock initialized: now 1786213747189456 us; error 56 us; skew 500 ppm
Aug 08 18:29:07 dist-test-slave-gp6n krb5kdc[5866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213747, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:07.192484 6290 init.cc:377] Logged in from keytab as oryx/127.29.78.131@KRBTEST.COM (short username oryx)
I20260808 18:29:07.193379 6290 webserver.cc:460] Webserver started at http://127.29.78.131:41793/ using document root <none> and password file <none>
I20260808 18:29:07.193934 6290 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:07.194042 6290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:07.194272 6290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:07.196895 6290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b"
format_stamp: "Formatted at 2026-08-08 18:29:07 on dist-test-slave-gp6n"
server_key: "ba949d2c24160986f400f04434137dd9"
server_key_iv: "323e71d5d48c4e8d4b653b0c7dbfaa53"
server_key_version: "encryptionkey@0"
I20260808 18:29:07.197481 6290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b"
format_stamp: "Formatted at 2026-08-08 18:29:07 on dist-test-slave-gp6n"
server_key: "ba949d2c24160986f400f04434137dd9"
server_key_iv: "323e71d5d48c4e8d4b653b0c7dbfaa53"
server_key_version: "encryptionkey@0"
I20260808 18:29:07.201191 6290 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260808 18:29:07.203650 6307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:07.204859 6290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:29:07.204993 6290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b"
format_stamp: "Formatted at 2026-08-08 18:29:07 on dist-test-slave-gp6n"
server_key: "ba949d2c24160986f400f04434137dd9"
server_key_iv: "323e71d5d48c4e8d4b653b0c7dbfaa53"
server_key_version: "encryptionkey@0"
I20260808 18:29:07.205111 6290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:07.225416 6290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:07.228771 6290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:07.228987 6290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:07.230762 6290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:07.230842 6290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:07.230903 6290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:07.230938 6290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 18:29:07 dist-test-slave-gp6n krb5kdc[5866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213747, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260808 18:29:07.243448 6290 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:36745
I20260808 18:29:07.244393 6290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:29:07.245217 6433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:36745 every 8 connection(s)
I20260808 18:29:07.246230 6318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.232046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42985 (local address 127.29.78.131:60751)
0808 18:29:07.232686 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:07.232731 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:07.233588 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:07.234163 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:07.234175 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:07.234727 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:07.235196 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.235210 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.236186 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.236189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.237273 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.237284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.237909 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.237913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.238088 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.238695 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:07.238711 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:07.241106 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:07.243383 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:07.243389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:07.243405 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:07.243744 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:07.244038 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:07.244043 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:07.244045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:07.244195 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:07.244659 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:07.244665 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:07.244952 (+ 287us) client_negotiation.cc:769] Sending connection context
0808 18:29:07.245148 (+ 196us) client_negotiation.cc:240] Negotiation successful
0808 18:29:07.245483 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":143,"threads_started":1}
I20260808 18:29:07.246743 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.231471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:60751 (local address 127.29.78.190:42985)
0808 18:29:07.231602 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.231609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.232849 (+ 1240us) server_negotiation.cc:407] Connection header received
0808 18:29:07.233804 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.233808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.233878 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.233960 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:07.235348 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.236074 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.237437 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.237749 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.237877 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.241280 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:07.241289 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:07.241296 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:07.241333 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:07.243268 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:07.243849 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:07.243853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:07.243855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:07.243944 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:07.244294 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:07.244299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:07.244300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:07.244548 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:07.244663 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:07.246454 (+ 1791us) server_negotiation.cc:299] Negotiation successful
0808 18:29:07.246594 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 18:29:07.247370 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 6290
I20260808 18:29:07.247516 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:29:07.247822 30010 external_mini_cluster.cc:1428] Setting key 90beb7060e3c23acde2ada6e1e3957f3
I20260808 18:29:07.248142 6435 heartbeater.cc:347] Connected to a master server at 127.29.78.190:42985
I20260808 18:29:07.248291 6435 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:07.249078 6435 heartbeater.cc:511] Master 127.29.78.190:42985 requested a full tablet report, sending...
I20260808 18:29:07.250186 5914 ts_manager.cc:194] Registered new tserver with Master: ac5d8094f08e4c4597eedb4b2a2ff27b (127.29.78.131:36745)
I20260808 18:29:07.250939 5914 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:60751
I20260808 18:29:07.256675 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.250367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:59801 (local address 127.29.78.190:42985)
0808 18:29:07.250525 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.250537 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.250551 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:29:07.250584 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.250587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.250625 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.250715 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:07.251079 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.251747 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.252594 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.252818 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.252923 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.253866 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:07.253870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:07.253873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:07.253896 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:07.255111 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:07.255526 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:07.255530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:07.255532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:07.255611 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:07.255886 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:07.255891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:07.255896 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:07.256065 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:07.256170 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:07.256437 (+ 267us) server_negotiation.cc:299] Negotiation successful
0808 18:29:07.256536 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 18:29:07.256685 6441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.250115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42985 (local address 127.29.78.131:59801)
0808 18:29:07.250351 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:07.250365 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:07.250483 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:07.250720 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:07.250722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:07.250875 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:07.250968 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.250972 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.251871 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.251874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.252470 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.252477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.252927 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.252930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.253018 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.253435 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:07.253446 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:07.253737 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:07.255219 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:07.255224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:07.255226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:07.255425 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:07.255709 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:07.255712 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:07.255714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:07.255798 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:07.256161 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:07.256164 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:07.256260 (+ 96us) client_negotiation.cc:769] Sending connection context
0808 18:29:07.256362 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 18:29:07.256495 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":84,"threads_started":1}
I20260808 18:29:07.261482 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 18:29:07 dist-test-slave-gp6n krb5kdc[5866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213747, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 18:29:07 dist-test-slave-gp6n krb5kdc[5866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:07 dist-test-slave-gp6n krb5kdc[5866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
W20260808 18:29:07.286819 5966 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:07.279912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41804 (local address 127.29.78.190:42985)
0808 18:29:07.280073 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.280077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.280090 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:29:07.280133 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.280135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.280173 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.280249 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:07.281046 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.281556 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.282440 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.282665 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.282745 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.286628 (+ 3883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41804: BlockingRecv error: recv got EOF from 127.0.0.1:41804 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 08 18:29:07 dist-test-slave-gp6n krb5kdc[5866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213747, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260808 18:29:07.297000 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.288315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41808 (local address 127.29.78.190:42985)
0808 18:29:07.288461 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.288464 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.288537 (+ 73us) server_negotiation.cc:407] Connection header received
0808 18:29:07.288680 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.288683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.288725 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.288799 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:07.289430 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.289938 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.290856 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.291264 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.291381 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.293760 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:07.293763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:07.293766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:07.293791 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:07.295301 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:07.295715 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:07.295718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:07.295719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:07.295798 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:07.296088 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:07.296090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:07.296091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:07.296262 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:07.296407 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:07.296684 (+ 277us) server_negotiation.cc:299] Negotiation successful
0808 18:29:07.296788 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 18:29:07.300011 5914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:29:07.303174 5914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:29:07.318470 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.312056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:42051 (local address 127.0.0.1:35376)
0808 18:29:07.312991 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:07.313006 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:07.313095 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:07.314124 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:07.314127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:07.314143 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:07.314297 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.314308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.315686 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.315689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.317065 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.317074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.317847 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.317850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.317954 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.317992 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 18:29:07.318119 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 18:29:07.318244 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":139,"thread_start_us":90,"threads_started":1}
I20260808 18:29:07.318687 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.312034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.29.78.129:39945)
0808 18:29:07.312428 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.312436 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.312847 (+ 411us) server_negotiation.cc:407] Connection header received
0808 18:29:07.313155 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.313161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.313375 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.313504 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:07.314000 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.315244 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.317174 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.318015 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.318137 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.318275 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:07.318386 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 18:29:07.318537 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":92,"threads_started":1}
I20260808 18:29:07.318930 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.312250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35376 (local address 127.29.78.130:42051)
0808 18:29:07.313558 (+ 1308us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.313571 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.313591 (+ 20us) server_negotiation.cc:407] Connection header received
0808 18:29:07.313657 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.313661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.313817 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.313963 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:07.314449 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.315557 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.317190 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.317718 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.317845 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.318015 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:07.318685 (+ 670us) server_negotiation.cc:299] Negotiation successful
0808 18:29:07.318797 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1174,"thread_start_us":103,"threads_started":1}
I20260808 18:29:07.318679 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.311871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:39945 (local address 127.0.0.1:35060)
0808 18:29:07.312707 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:07.312760 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:07.312965 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:07.313699 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:07.313703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:07.313723 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:07.313833 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.313838 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.315408 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.315411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.317036 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.317047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.318143 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.318146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.318260 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.318281 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 18:29:07.318353 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 18:29:07.318500 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":328,"thread_start_us":126,"threads_started":1}
I20260808 18:29:07.319516 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.312831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56150 (local address 127.29.78.131:36745)
0808 18:29:07.313242 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.313252 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.313505 (+ 253us) server_negotiation.cc:407] Connection header received
0808 18:29:07.313596 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.313603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.313795 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.313963 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:07.314258 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.315186 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.316926 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.317753 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.317868 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.318042 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:07.319241 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0808 18:29:07.319357 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":102,"threads_started":1}
I20260808 18:29:07.318688 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.312672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:36745 (local address 127.0.0.1:56150)
0808 18:29:07.313372 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:07.313385 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:07.313482 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:07.313982 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:07.313984 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:07.314002 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:07.314101 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.314107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.315336 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.315340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.316788 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.316798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.318117 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.318121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.318213 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.318276 (+ 63us) client_negotiation.cc:769] Sending connection context
0808 18:29:07.318365 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 18:29:07.318528 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":157,"thread_start_us":113,"threads_started":1}
I20260808 18:29:07.322209 6212 tablet_service.cc:1467] Processing CreateTablet for tablet 7cab8c83063b4c689f1fb23ddcde2401 (DEFAULT_TABLE table=test-table [id=003f28e7c69343babd5b6bce5f6368e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:29:07.322878 6063 tablet_service.cc:1467] Processing CreateTablet for tablet 7cab8c83063b4c689f1fb23ddcde2401 (DEFAULT_TABLE table=test-table [id=003f28e7c69343babd5b6bce5f6368e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:29:07.323464 6212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cab8c83063b4c689f1fb23ddcde2401. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:07.323910 6063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cab8c83063b4c689f1fb23ddcde2401. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:07.324298 6361 tablet_service.cc:1467] Processing CreateTablet for tablet 7cab8c83063b4c689f1fb23ddcde2401 (DEFAULT_TABLE table=test-table [id=003f28e7c69343babd5b6bce5f6368e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:29:07.325176 6361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cab8c83063b4c689f1fb23ddcde2401. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:07.331612 6467 tablet_bootstrap.cc:492] T 7cab8c83063b4c689f1fb23ddcde2401 P 1108a5faf487493bb7703e1cae80a1e2: Bootstrap starting.
I20260808 18:29:07.332661 6469 tablet_bootstrap.cc:492] T 7cab8c83063b4c689f1fb23ddcde2401 P ac5d8094f08e4c4597eedb4b2a2ff27b: Bootstrap starting.
I20260808 18:29:07.333350 6468 tablet_bootstrap.cc:492] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce: Bootstrap starting.
I20260808 18:29:07.334312 6467 tablet_bootstrap.cc:654] T 7cab8c83063b4c689f1fb23ddcde2401 P 1108a5faf487493bb7703e1cae80a1e2: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:07.334585 6469 tablet_bootstrap.cc:654] T 7cab8c83063b4c689f1fb23ddcde2401 P ac5d8094f08e4c4597eedb4b2a2ff27b: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:07.335335 6469 log.cc:824] T 7cab8c83063b4c689f1fb23ddcde2401 P ac5d8094f08e4c4597eedb4b2a2ff27b: Log is configured to *not* fsync() on all Append() calls
I20260808 18:29:07.335570 6468 tablet_bootstrap.cc:654] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:07.335669 6467 log.cc:824] T 7cab8c83063b4c689f1fb23ddcde2401 P 1108a5faf487493bb7703e1cae80a1e2: Log is configured to *not* fsync() on all Append() calls
I20260808 18:29:07.336428 6468 log.cc:824] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce: Log is configured to *not* fsync() on all Append() calls
I20260808 18:29:07.337648 6467 tablet_bootstrap.cc:492] T 7cab8c83063b4c689f1fb23ddcde2401 P 1108a5faf487493bb7703e1cae80a1e2: No bootstrap required, opened a new log
I20260808 18:29:07.337854 6467 ts_tablet_manager.cc:1397] T 7cab8c83063b4c689f1fb23ddcde2401 P 1108a5faf487493bb7703e1cae80a1e2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:29:07.337858 6469 tablet_bootstrap.cc:492] T 7cab8c83063b4c689f1fb23ddcde2401 P ac5d8094f08e4c4597eedb4b2a2ff27b: No bootstrap required, opened a new log
I20260808 18:29:07.338058 6469 ts_tablet_manager.cc:1397] T 7cab8c83063b4c689f1fb23ddcde2401 P ac5d8094f08e4c4597eedb4b2a2ff27b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260808 18:29:07.338392 6468 tablet_bootstrap.cc:492] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce: No bootstrap required, opened a new log
I20260808 18:29:07.338641 6468 ts_tablet_manager.cc:1397] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260808 18:29:07.340955 6467 raft_consensus.cc:348] T 7cab8c83063b4c689f1fb23ddcde2401 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } }
I20260808 18:29:07.341465 6467 raft_consensus.cc:374] T 7cab8c83063b4c689f1fb23ddcde2401 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:07.341526 6467 raft_consensus.cc:729] T 7cab8c83063b4c689f1fb23ddcde2401 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1108a5faf487493bb7703e1cae80a1e2, State: Initialized, Role: FOLLOWER
I20260808 18:29:07.342002 6467 consensus_queue.cc:260] T 7cab8c83063b4c689f1fb23ddcde2401 P 1108a5faf487493bb7703e1cae80a1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } }
I20260808 18:29:07.342720 6280 heartbeater.cc:503] Master 127.29.78.190:42985 was elected leader, sending a full tablet report...
I20260808 18:29:07.342552 6469 raft_consensus.cc:348] T 7cab8c83063b4c689f1fb23ddcde2401 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } }
I20260808 18:29:07.342813 6469 raft_consensus.cc:374] T 7cab8c83063b4c689f1fb23ddcde2401 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:07.342880 6469 raft_consensus.cc:729] T 7cab8c83063b4c689f1fb23ddcde2401 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac5d8094f08e4c4597eedb4b2a2ff27b, State: Initialized, Role: FOLLOWER
I20260808 18:29:07.343116 6468 raft_consensus.cc:348] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } }
I20260808 18:29:07.343372 6468 raft_consensus.cc:374] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:07.343327 6469 consensus_queue.cc:260] T 7cab8c83063b4c689f1fb23ddcde2401 P ac5d8094f08e4c4597eedb4b2a2ff27b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } }
I20260808 18:29:07.343441 6468 raft_consensus.cc:729] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b992bf381f0a41849ffc437f30c71bce, State: Initialized, Role: FOLLOWER
I20260808 18:29:07.343983 6435 heartbeater.cc:503] Master 127.29.78.190:42985 was elected leader, sending a full tablet report...
I20260808 18:29:07.344228 6467 ts_tablet_manager.cc:1428] T 7cab8c83063b4c689f1fb23ddcde2401 P 1108a5faf487493bb7703e1cae80a1e2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 18:29:07.344426 6469 ts_tablet_manager.cc:1428] T 7cab8c83063b4c689f1fb23ddcde2401 P ac5d8094f08e4c4597eedb4b2a2ff27b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 18:29:07.344864 6468 consensus_queue.cc:260] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } }
I20260808 18:29:07.345541 6132 heartbeater.cc:503] Master 127.29.78.190:42985 was elected leader, sending a full tablet report...
I20260808 18:29:07.345852 6468 ts_tablet_manager.cc:1428] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260808 18:29:07.356168 6476 raft_consensus.cc:482] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:29:07.356374 6476 raft_consensus.cc:504] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } }
I20260808 18:29:07.357622 6476 leader_election.cc:288] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac5d8094f08e4c4597eedb4b2a2ff27b (127.29.78.131:36745), 1108a5faf487493bb7703e1cae80a1e2 (127.29.78.130:42051)
I20260808 18:29:07.363083 6478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.357880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:42051 (local address 127.29.78.129:34539)
0808 18:29:07.358268 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:07.358281 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:07.358432 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:07.358670 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:07.358672 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:07.358710 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:07.358939 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.358946 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.359976 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.359979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.361489 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.361500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.362534 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.362539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.362670 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.362698 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 18:29:07.362786 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 18:29:07.362879 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":175,"spinlock_wait_cycles":7040,"thread_start_us":129,"threads_started":1}
I20260808 18:29:07.363169 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.358019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:34539 (local address 127.29.78.130:42051)
0808 18:29:07.358209 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.358220 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.358343 (+ 123us) server_negotiation.cc:407] Connection header received
0808 18:29:07.358523 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.358534 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.358584 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.358657 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:07.359060 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.359864 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.361692 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.362300 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.362462 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.362520 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:07.362854 (+ 334us) server_negotiation.cc:299] Negotiation successful
0808 18:29:07.362971 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260808 18:29:07.363749 6234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cab8c83063b4c689f1fb23ddcde2401" candidate_uuid: "b992bf381f0a41849ffc437f30c71bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1108a5faf487493bb7703e1cae80a1e2" is_pre_election: true
I20260808 18:29:07.364084 6234 raft_consensus.cc:2454] T 7cab8c83063b4c689f1fb23ddcde2401 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b992bf381f0a41849ffc437f30c71bce in term 0.
I20260808 18:29:07.364596 6009 leader_election.cc:302] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1108a5faf487493bb7703e1cae80a1e2, b992bf381f0a41849ffc437f30c71bce; no voters:
I20260808 18:29:07.364792 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.357949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:58477 (local address 127.29.78.131:36745)
0808 18:29:07.358090 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.358094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.358298 (+ 204us) server_negotiation.cc:407] Connection header received
0808 18:29:07.358482 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.358487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.358538 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.358626 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:07.359063 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.359861 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.362579 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.363215 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.363339 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.363386 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:07.364593 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0808 18:29:07.364669 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 18:29:07.364800 6477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.357871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:36745 (local address 127.29.78.129:58477)
0808 18:29:07.358214 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:07.358230 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:07.358354 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:07.358665 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:07.358668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:07.358709 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:07.358939 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.358946 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.359976 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.359979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.361488 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.361500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.363448 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.363451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.364450 (+ 999us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.364476 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 18:29:07.364558 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 18:29:07.364632 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":153,"spinlock_wait_cycles":26240,"thread_start_us":145,"threads_started":1}
I20260808 18:29:07.365283 6476 raft_consensus.cc:2781] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:29:07.365392 6476 raft_consensus.cc:482] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:29:07.365439 6476 raft_consensus.cc:3037] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:07.365680 6383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cab8c83063b4c689f1fb23ddcde2401" candidate_uuid: "b992bf381f0a41849ffc437f30c71bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" is_pre_election: true
I20260808 18:29:07.365988 6383 raft_consensus.cc:2454] T 7cab8c83063b4c689f1fb23ddcde2401 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b992bf381f0a41849ffc437f30c71bce in term 0.
I20260808 18:29:07.366757 6476 raft_consensus.cc:504] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } }
I20260808 18:29:07.367208 6476 leader_election.cc:288] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce [CANDIDATE]: Term 1 election: Requested vote from peers ac5d8094f08e4c4597eedb4b2a2ff27b (127.29.78.131:36745), 1108a5faf487493bb7703e1cae80a1e2 (127.29.78.130:42051)
I20260808 18:29:07.367446 6383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cab8c83063b4c689f1fb23ddcde2401" candidate_uuid: "b992bf381f0a41849ffc437f30c71bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b"
I20260808 18:29:07.367583 6383 raft_consensus.cc:3037] T 7cab8c83063b4c689f1fb23ddcde2401 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:07.368623 6383 raft_consensus.cc:2454] T 7cab8c83063b4c689f1fb23ddcde2401 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b992bf381f0a41849ffc437f30c71bce in term 1.
I20260808 18:29:07.368923 6010 leader_election.cc:302] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac5d8094f08e4c4597eedb4b2a2ff27b, b992bf381f0a41849ffc437f30c71bce; no voters:
I20260808 18:29:07.369132 6476 raft_consensus.cc:2781] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:29:07.369385 6476 raft_consensus.cc:686] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce [term 1 LEADER]: Becoming Leader. State: Replica: b992bf381f0a41849ffc437f30c71bce, State: Running, Role: LEADER
I20260808 18:29:07.369706 6476 consensus_queue.cc:237] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } }
I20260808 18:29:07.373190 5913 catalog_manager.cc:5495] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce reported cstate change: term changed from 0 to 1, leader changed from <none> to b992bf381f0a41849ffc437f30c71bce (127.29.78.129). New cstate: current_term: 1 leader_uuid: "b992bf381f0a41849ffc437f30c71bce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } health_report { overall_health: UNKNOWN } } }
I20260808 18:29:07.378301 6234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cab8c83063b4c689f1fb23ddcde2401" candidate_uuid: "b992bf381f0a41849ffc437f30c71bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1108a5faf487493bb7703e1cae80a1e2"
I20260808 18:29:07.378465 6234 raft_consensus.cc:3037] T 7cab8c83063b4c689f1fb23ddcde2401 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:07.379814 6234 raft_consensus.cc:2454] T 7cab8c83063b4c689f1fb23ddcde2401 P 1108a5faf487493bb7703e1cae80a1e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b992bf381f0a41849ffc437f30c71bce in term 1.
I20260808 18:29:07.398595 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.394928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35068 (local address 127.29.78.129:39945)
0808 18:29:07.395073 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.395078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.395092 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:29:07.395274 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.395277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.395333 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.395498 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:29:07.395749 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.396260 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.397138 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.397362 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.397481 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.397764 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:29:07.398183 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:29:07.398323 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:07.398364 (+ 41us) server_negotiation.cc:299] Negotiation successful
0808 18:29:07.398455 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 18:29:07.404310 6234 raft_consensus.cc:1261] T 7cab8c83063b4c689f1fb23ddcde2401 P 1108a5faf487493bb7703e1cae80a1e2 [term 1 FOLLOWER]: Refusing update from remote peer b992bf381f0a41849ffc437f30c71bce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:29:07.404665 6383 raft_consensus.cc:1261] T 7cab8c83063b4c689f1fb23ddcde2401 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 1 FOLLOWER]: Refusing update from remote peer b992bf381f0a41849ffc437f30c71bce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:29:07.405015 6476 consensus_queue.cc:1035] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:29:07.405316 6479 consensus_queue.cc:1035] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:29:07.413134 6484 mvcc.cc:205] Tried to move back new op lower bound from 7316331509361270784 to 7316331509229092864. Current Snapshot: MvccSnapshot[applied={T|T < 7316331509361270784}]
I20260808 18:29:07.423166 5913 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:29:07.423348 5913 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:29:07.425704 5913 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad: Sending DeleteTablet for 3 replicas of tablet 7cab8c83063b4c689f1fb23ddcde2401
I20260808 18:29:07.426465 6063 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cab8c83063b4c689f1fb23ddcde2401 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:29:07 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:35060
I20260808 18:29:07.426517 6361 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cab8c83063b4c689f1fb23ddcde2401 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:29:07 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:56150
I20260808 18:29:07.426717 6212 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cab8c83063b4c689f1fb23ddcde2401 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:29:07 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:35376
I20260808 18:29:07.427198 6495 tablet_replica.cc:330] T 7cab8c83063b4c689f1fb23ddcde2401 P 1108a5faf487493bb7703e1cae80a1e2: stopping tablet replica
I20260808 18:29:07.427210 6492 tablet_replica.cc:330] T 7cab8c83063b4c689f1fb23ddcde2401 P ac5d8094f08e4c4597eedb4b2a2ff27b: stopping tablet replica
I20260808 18:29:07.427501 6492 raft_consensus.cc:2229] T 7cab8c83063b4c689f1fb23ddcde2401 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:29:07.427510 6495 raft_consensus.cc:2229] T 7cab8c83063b4c689f1fb23ddcde2401 P 1108a5faf487493bb7703e1cae80a1e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:29:07.427819 6495 raft_consensus.cc:2258] T 7cab8c83063b4c689f1fb23ddcde2401 P 1108a5faf487493bb7703e1cae80a1e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:29:07.427830 6492 raft_consensus.cc:2258] T 7cab8c83063b4c689f1fb23ddcde2401 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:29:07.427778 5913 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 18:29:07.429440 6492 ts_tablet_manager.cc:1905] T 7cab8c83063b4c689f1fb23ddcde2401 P ac5d8094f08e4c4597eedb4b2a2ff27b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:29:07.430318 6495 ts_tablet_manager.cc:1905] T 7cab8c83063b4c689f1fb23ddcde2401 P 1108a5faf487493bb7703e1cae80a1e2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 18:29:07.430647 5913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:29:07.432585 6492 ts_tablet_manager.cc:1918] T 7cab8c83063b4c689f1fb23ddcde2401 P ac5d8094f08e4c4597eedb4b2a2ff27b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:29:07.432682 6492 log.cc:1196] T 7cab8c83063b4c689f1fb23ddcde2401 P ac5d8094f08e4c4597eedb4b2a2ff27b: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/7cab8c83063b4c689f1fb23ddcde2401
I20260808 18:29:07.433007 6492 ts_tablet_manager.cc:1939] T 7cab8c83063b4c689f1fb23ddcde2401 P ac5d8094f08e4c4597eedb4b2a2ff27b: Deleting consensus metadata
I20260808 18:29:07.433753 6495 ts_tablet_manager.cc:1918] T 7cab8c83063b4c689f1fb23ddcde2401 P 1108a5faf487493bb7703e1cae80a1e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:29:07.433840 6495 log.cc:1196] T 7cab8c83063b4c689f1fb23ddcde2401 P 1108a5faf487493bb7703e1cae80a1e2: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/7cab8c83063b4c689f1fb23ddcde2401
I20260808 18:29:07.434172 6495 ts_tablet_manager.cc:1939] T 7cab8c83063b4c689f1fb23ddcde2401 P 1108a5faf487493bb7703e1cae80a1e2: Deleting consensus metadata
I20260808 18:29:07.434182 5900 catalog_manager.cc:4841] TS ac5d8094f08e4c4597eedb4b2a2ff27b (127.29.78.131:36745): tablet 7cab8c83063b4c689f1fb23ddcde2401 (table test-table [id=003f28e7c69343babd5b6bce5f6368e7]) successfully deleted
I20260808 18:29:07.434502 6494 tablet_replica.cc:330] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce: stopping tablet replica
I20260808 18:29:07.434787 6494 raft_consensus.cc:2229] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:29:07.435173 6494 raft_consensus.cc:2258] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:29:07.435321 5899 catalog_manager.cc:4841] TS 1108a5faf487493bb7703e1cae80a1e2 (127.29.78.130:42051): tablet 7cab8c83063b4c689f1fb23ddcde2401 (table test-table [id=003f28e7c69343babd5b6bce5f6368e7]) successfully deleted
I20260808 18:29:07.436434 6494 ts_tablet_manager.cc:1905] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:29:07.436883 6063 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac5b0029f8c478e95853c7cbebe53d6 (DEFAULT_TABLE table=test-table [id=2bb430669e354a02a113c656abac9f24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:29:07.437158 6063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ac5b0029f8c478e95853c7cbebe53d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:07.437131 6212 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac5b0029f8c478e95853c7cbebe53d6 (DEFAULT_TABLE table=test-table [id=2bb430669e354a02a113c656abac9f24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:29:07.437343 6212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ac5b0029f8c478e95853c7cbebe53d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:07.438977 6494 ts_tablet_manager.cc:1918] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:29:07.439062 6494 log.cc:1196] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-0/wal/wals/7cab8c83063b4c689f1fb23ddcde2401
I20260808 18:29:07.439455 6468 tablet_bootstrap.cc:492] T 0ac5b0029f8c478e95853c7cbebe53d6 P b992bf381f0a41849ffc437f30c71bce: Bootstrap starting.
I20260808 18:29:07.439539 6467 tablet_bootstrap.cc:492] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2: Bootstrap starting.
I20260808 18:29:07.440150 6361 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac5b0029f8c478e95853c7cbebe53d6 (DEFAULT_TABLE table=test-table [id=2bb430669e354a02a113c656abac9f24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:29:07.440357 6468 tablet_bootstrap.cc:654] T 0ac5b0029f8c478e95853c7cbebe53d6 P b992bf381f0a41849ffc437f30c71bce: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:07.440392 6361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ac5b0029f8c478e95853c7cbebe53d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:07.440507 6467 tablet_bootstrap.cc:654] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:07.440855 6494 ts_tablet_manager.cc:1939] T 7cab8c83063b4c689f1fb23ddcde2401 P b992bf381f0a41849ffc437f30c71bce: Deleting consensus metadata
I20260808 18:29:07.441922 5900 catalog_manager.cc:4841] TS b992bf381f0a41849ffc437f30c71bce (127.29.78.129:39945): tablet 7cab8c83063b4c689f1fb23ddcde2401 (table test-table [id=003f28e7c69343babd5b6bce5f6368e7]) successfully deleted
I20260808 18:29:07.442553 6468 tablet_bootstrap.cc:492] T 0ac5b0029f8c478e95853c7cbebe53d6 P b992bf381f0a41849ffc437f30c71bce: No bootstrap required, opened a new log
I20260808 18:29:07.442646 6469 tablet_bootstrap.cc:492] T 0ac5b0029f8c478e95853c7cbebe53d6 P ac5d8094f08e4c4597eedb4b2a2ff27b: Bootstrap starting.
I20260808 18:29:07.442641 6468 ts_tablet_manager.cc:1397] T 0ac5b0029f8c478e95853c7cbebe53d6 P b992bf381f0a41849ffc437f30c71bce: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 18:29:07.443071 6468 raft_consensus.cc:348] T 0ac5b0029f8c478e95853c7cbebe53d6 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.443208 6468 raft_consensus.cc:374] T 0ac5b0029f8c478e95853c7cbebe53d6 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:07.443245 6468 raft_consensus.cc:729] T 0ac5b0029f8c478e95853c7cbebe53d6 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b992bf381f0a41849ffc437f30c71bce, State: Initialized, Role: FOLLOWER
I20260808 18:29:07.443356 6468 consensus_queue.cc:260] T 0ac5b0029f8c478e95853c7cbebe53d6 P b992bf381f0a41849ffc437f30c71bce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.443601 6469 tablet_bootstrap.cc:654] T 0ac5b0029f8c478e95853c7cbebe53d6 P ac5d8094f08e4c4597eedb4b2a2ff27b: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:07.443647 6468 ts_tablet_manager.cc:1428] T 0ac5b0029f8c478e95853c7cbebe53d6 P b992bf381f0a41849ffc437f30c71bce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:29:07.444012 6467 tablet_bootstrap.cc:492] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2: No bootstrap required, opened a new log
I20260808 18:29:07.444082 6467 ts_tablet_manager.cc:1397] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260808 18:29:07.444471 6467 raft_consensus.cc:348] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.444566 6467 raft_consensus.cc:374] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:07.444595 6467 raft_consensus.cc:729] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1108a5faf487493bb7703e1cae80a1e2, State: Initialized, Role: FOLLOWER
I20260808 18:29:07.444712 6467 consensus_queue.cc:260] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.444984 6467 ts_tablet_manager.cc:1428] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:29:07.445107 6469 tablet_bootstrap.cc:492] T 0ac5b0029f8c478e95853c7cbebe53d6 P ac5d8094f08e4c4597eedb4b2a2ff27b: No bootstrap required, opened a new log
I20260808 18:29:07.445189 6469 ts_tablet_manager.cc:1397] T 0ac5b0029f8c478e95853c7cbebe53d6 P ac5d8094f08e4c4597eedb4b2a2ff27b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:29:07.445649 6469 raft_consensus.cc:348] T 0ac5b0029f8c478e95853c7cbebe53d6 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.445756 6469 raft_consensus.cc:374] T 0ac5b0029f8c478e95853c7cbebe53d6 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:07.445791 6469 raft_consensus.cc:729] T 0ac5b0029f8c478e95853c7cbebe53d6 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac5d8094f08e4c4597eedb4b2a2ff27b, State: Initialized, Role: FOLLOWER
I20260808 18:29:07.445920 6469 consensus_queue.cc:260] T 0ac5b0029f8c478e95853c7cbebe53d6 P ac5d8094f08e4c4597eedb4b2a2ff27b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.446178 6469 ts_tablet_manager.cc:1428] T 0ac5b0029f8c478e95853c7cbebe53d6 P ac5d8094f08e4c4597eedb4b2a2ff27b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260808 18:29:07.495460 6436 tablet.cc:2369] T 0ac5b0029f8c478e95853c7cbebe53d6 P ac5d8094f08e4c4597eedb4b2a2ff27b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 18:29:07.510465 6281 tablet.cc:2369] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:29:07.518307 6474 raft_consensus.cc:482] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:29:07.518461 6474 raft_consensus.cc:504] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.519582 6474 leader_election.cc:288] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b992bf381f0a41849ffc437f30c71bce (127.29.78.129:39945), ac5d8094f08e4c4597eedb4b2a2ff27b (127.29.78.131:36745)
I20260808 18:29:07.524852 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.520053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:43359 (local address 127.29.78.129:39945)
0808 18:29:07.520192 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.520198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.520427 (+ 229us) server_negotiation.cc:407] Connection header received
0808 18:29:07.520667 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.520672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.520737 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.520780 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:07.521241 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.522044 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.523839 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.524246 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.524378 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.524433 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:07.524636 (+ 203us) server_negotiation.cc:299] Negotiation successful
0808 18:29:07.524717 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 18:29:07.525018 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.520639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:45331 (local address 127.29.78.131:36745)
0808 18:29:07.520768 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.520773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.520970 (+ 197us) server_negotiation.cc:407] Connection header received
0808 18:29:07.521113 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.521116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.521171 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.521246 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:07.521479 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.521964 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.523790 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.524428 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.524557 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.524610 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:07.524801 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 18:29:07.524888 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 18:29:07.525135 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.520306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:36745 (local address 127.29.78.130:45331)
0808 18:29:07.520854 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:07.520870 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:07.521004 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:07.521252 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:07.521255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:07.521269 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:07.521375 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.521379 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.522093 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.522096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.523643 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.523654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.524582 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.524588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.524681 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.524700 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 18:29:07.524770 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 18:29:07.524834 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":110,"spinlock_wait_cycles":768,"thread_start_us":72,"threads_started":1}
I20260808 18:29:07.525377 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.519959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:39945 (local address 127.29.78.130:43359)
0808 18:29:07.520338 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:07.520355 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:07.520510 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:07.520903 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:07.520905 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:07.520920 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:07.521112 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.521119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.522166 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.522169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.523645 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.523654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.524370 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.524373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.524477 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.524507 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 18:29:07.525059 (+ 552us) client_negotiation.cc:240] Negotiation successful
0808 18:29:07.525151 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":100,"spinlock_wait_cycles":23808,"thread_start_us":80,"threads_started":1}
I20260808 18:29:07.525871 6084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac5b0029f8c478e95853c7cbebe53d6" candidate_uuid: "1108a5faf487493bb7703e1cae80a1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b992bf381f0a41849ffc437f30c71bce" is_pre_election: true
I20260808 18:29:07.526119 6084 raft_consensus.cc:2454] T 0ac5b0029f8c478e95853c7cbebe53d6 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1108a5faf487493bb7703e1cae80a1e2 in term 0.
I20260808 18:29:07.526321 6383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac5b0029f8c478e95853c7cbebe53d6" candidate_uuid: "1108a5faf487493bb7703e1cae80a1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" is_pre_election: true
I20260808 18:29:07.526547 6383 raft_consensus.cc:2454] T 0ac5b0029f8c478e95853c7cbebe53d6 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1108a5faf487493bb7703e1cae80a1e2 in term 0.
I20260808 18:29:07.526543 6162 leader_election.cc:302] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1108a5faf487493bb7703e1cae80a1e2, b992bf381f0a41849ffc437f30c71bce; no voters:
I20260808 18:29:07.526835 6474 raft_consensus.cc:2781] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:29:07.526894 6474 raft_consensus.cc:482] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:29:07.526933 6474 raft_consensus.cc:3037] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:07.527922 6474 raft_consensus.cc:504] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.528244 6474 leader_election.cc:288] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2 [CANDIDATE]: Term 1 election: Requested vote from peers b992bf381f0a41849ffc437f30c71bce (127.29.78.129:39945), ac5d8094f08e4c4597eedb4b2a2ff27b (127.29.78.131:36745)
I20260808 18:29:07.528538 6084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac5b0029f8c478e95853c7cbebe53d6" candidate_uuid: "1108a5faf487493bb7703e1cae80a1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b992bf381f0a41849ffc437f30c71bce"
I20260808 18:29:07.528600 6383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac5b0029f8c478e95853c7cbebe53d6" candidate_uuid: "1108a5faf487493bb7703e1cae80a1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b"
I20260808 18:29:07.528683 6084 raft_consensus.cc:3037] T 0ac5b0029f8c478e95853c7cbebe53d6 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:07.528739 6383 raft_consensus.cc:3037] T 0ac5b0029f8c478e95853c7cbebe53d6 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:07.529625 6383 raft_consensus.cc:2454] T 0ac5b0029f8c478e95853c7cbebe53d6 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1108a5faf487493bb7703e1cae80a1e2 in term 1.
I20260808 18:29:07.529897 6162 leader_election.cc:302] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1108a5faf487493bb7703e1cae80a1e2, ac5d8094f08e4c4597eedb4b2a2ff27b; no voters:
I20260808 18:29:07.529942 6084 raft_consensus.cc:2454] T 0ac5b0029f8c478e95853c7cbebe53d6 P b992bf381f0a41849ffc437f30c71bce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1108a5faf487493bb7703e1cae80a1e2 in term 1.
I20260808 18:29:07.530092 6474 raft_consensus.cc:2781] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:29:07.530278 6474 raft_consensus.cc:686] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2 [term 1 LEADER]: Becoming Leader. State: Replica: 1108a5faf487493bb7703e1cae80a1e2, State: Running, Role: LEADER
I20260808 18:29:07.530550 6474 consensus_queue.cc:237] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.533092 5915 catalog_manager.cc:5495] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1108a5faf487493bb7703e1cae80a1e2 (127.29.78.130). New cstate: current_term: 1 leader_uuid: "1108a5faf487493bb7703e1cae80a1e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } health_report { overall_health: UNKNOWN } } }
W20260808 18:29:07.543068 6133 tablet.cc:2369] T 0ac5b0029f8c478e95853c7cbebe53d6 P b992bf381f0a41849ffc437f30c71bce: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 18:29:07 dist-test-slave-gp6n krb5kdc[5866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213746, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.190@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260808 18:29:07.568944 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.560334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41824 (local address 127.29.78.190:42985)
0808 18:29:07.560485 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.560490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.560619 (+ 129us) server_negotiation.cc:407] Connection header received
0808 18:29:07.560775 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.560779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.560824 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.560912 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:07.561587 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.562115 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.563065 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.563278 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.563389 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.565686 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:07.565691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:07.565699 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:07.565719 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:07.567205 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:07.567678 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:07.567684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:07.567685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:07.567765 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:07.568040 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:07.568043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:07.568045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:07.568219 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:07.568325 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:07.568620 (+ 295us) server_negotiation.cc:299] Negotiation successful
0808 18:29:07.568816 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":74}
I20260808 18:29:07.568953 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.560230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42985 (local address 127.0.0.1:41824)
0808 18:29:07.560506 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:07.560522 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:07.560675 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:07.560934 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:07.560937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:07.561353 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:07.561444 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.561451 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.562222 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.562229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.562926 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.562937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.563389 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.563393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.563469 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.564072 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:07.564085 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:07.565551 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:07.567315 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:07.567327 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:07.567333 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:07.567578 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:07.567855 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:07.567860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:07.567862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:07.567945 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:07.568325 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:07.568333 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:07.568447 (+ 114us) client_negotiation.cc:769] Sending connection context
0808 18:29:07.568575 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 18:29:07.568747 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":94,"threads_started":1}
I20260808 18:29:07.573683 5915 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:41824:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 18:29:07.574688 5915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:29:07.579627 6212 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad80bf3a8e04366b1829a0662a9ad92 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4649d6060def4b0b8876c4f55e4d02a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 18:29:07.579635 6063 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad80bf3a8e04366b1829a0662a9ad92 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4649d6060def4b0b8876c4f55e4d02a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 18:29:07.580020 6063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad80bf3a8e04366b1829a0662a9ad92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:07.580020 6212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad80bf3a8e04366b1829a0662a9ad92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:07.582854 6468 tablet_bootstrap.cc:492] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce: Bootstrap starting.
I20260808 18:29:07.582903 6467 tablet_bootstrap.cc:492] T 5ad80bf3a8e04366b1829a0662a9ad92 P 1108a5faf487493bb7703e1cae80a1e2: Bootstrap starting.
I20260808 18:29:07.583024 6361 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad80bf3a8e04366b1829a0662a9ad92 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4649d6060def4b0b8876c4f55e4d02a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 18:29:07.583418 6361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad80bf3a8e04366b1829a0662a9ad92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:07.583765 6468 tablet_bootstrap.cc:654] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:07.583765 6467 tablet_bootstrap.cc:654] T 5ad80bf3a8e04366b1829a0662a9ad92 P 1108a5faf487493bb7703e1cae80a1e2: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:07.585156 6468 tablet_bootstrap.cc:492] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce: No bootstrap required, opened a new log
I20260808 18:29:07.585247 6468 ts_tablet_manager.cc:1397] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 18:29:07.585664 6468 raft_consensus.cc:348] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.585749 6468 raft_consensus.cc:374] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:07.585770 6468 raft_consensus.cc:729] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b992bf381f0a41849ffc437f30c71bce, State: Initialized, Role: FOLLOWER
I20260808 18:29:07.585898 6468 consensus_queue.cc:260] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.586200 6468 ts_tablet_manager.cc:1428] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:29:07.586584 6469 tablet_bootstrap.cc:492] T 5ad80bf3a8e04366b1829a0662a9ad92 P ac5d8094f08e4c4597eedb4b2a2ff27b: Bootstrap starting.
I20260808 18:29:07.586534 6487 tablet_replica.cc:439] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } } }
I20260808 18:29:07.586661 6487 tablet_replica.cc:442] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:29:07.587167 6467 tablet_bootstrap.cc:492] T 5ad80bf3a8e04366b1829a0662a9ad92 P 1108a5faf487493bb7703e1cae80a1e2: No bootstrap required, opened a new log
I20260808 18:29:07.587257 6467 ts_tablet_manager.cc:1397] T 5ad80bf3a8e04366b1829a0662a9ad92 P 1108a5faf487493bb7703e1cae80a1e2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260808 18:29:07.587657 6469 tablet_bootstrap.cc:654] T 5ad80bf3a8e04366b1829a0662a9ad92 P ac5d8094f08e4c4597eedb4b2a2ff27b: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:07.587734 6467 raft_consensus.cc:348] T 5ad80bf3a8e04366b1829a0662a9ad92 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.587860 6467 raft_consensus.cc:374] T 5ad80bf3a8e04366b1829a0662a9ad92 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:07.587898 6467 raft_consensus.cc:729] T 5ad80bf3a8e04366b1829a0662a9ad92 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1108a5faf487493bb7703e1cae80a1e2, State: Initialized, Role: FOLLOWER
I20260808 18:29:07.588011 6467 consensus_queue.cc:260] T 5ad80bf3a8e04366b1829a0662a9ad92 P 1108a5faf487493bb7703e1cae80a1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.588279 6467 ts_tablet_manager.cc:1428] T 5ad80bf3a8e04366b1829a0662a9ad92 P 1108a5faf487493bb7703e1cae80a1e2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 18:29:07.588251 6474 tablet_replica.cc:439] T 5ad80bf3a8e04366b1829a0662a9ad92 P 1108a5faf487493bb7703e1cae80a1e2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } } }
I20260808 18:29:07.588393 6474 tablet_replica.cc:442] T 5ad80bf3a8e04366b1829a0662a9ad92 P 1108a5faf487493bb7703e1cae80a1e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:29:07.589272 6469 tablet_bootstrap.cc:492] T 5ad80bf3a8e04366b1829a0662a9ad92 P ac5d8094f08e4c4597eedb4b2a2ff27b: No bootstrap required, opened a new log
I20260808 18:29:07.589382 6469 ts_tablet_manager.cc:1397] T 5ad80bf3a8e04366b1829a0662a9ad92 P ac5d8094f08e4c4597eedb4b2a2ff27b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:29:07.589844 6469 raft_consensus.cc:348] T 5ad80bf3a8e04366b1829a0662a9ad92 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.589960 6469 raft_consensus.cc:374] T 5ad80bf3a8e04366b1829a0662a9ad92 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:07.590004 6469 raft_consensus.cc:729] T 5ad80bf3a8e04366b1829a0662a9ad92 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac5d8094f08e4c4597eedb4b2a2ff27b, State: Initialized, Role: FOLLOWER
I20260808 18:29:07.590128 6469 consensus_queue.cc:260] T 5ad80bf3a8e04366b1829a0662a9ad92 P ac5d8094f08e4c4597eedb4b2a2ff27b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.590441 6469 ts_tablet_manager.cc:1428] T 5ad80bf3a8e04366b1829a0662a9ad92 P ac5d8094f08e4c4597eedb4b2a2ff27b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260808 18:29:07.590378 6475 tablet_replica.cc:439] T 5ad80bf3a8e04366b1829a0662a9ad92 P ac5d8094f08e4c4597eedb4b2a2ff27b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } } }
I20260808 18:29:07.590551 6475 tablet_replica.cc:442] T 5ad80bf3a8e04366b1829a0662a9ad92 P ac5d8094f08e4c4597eedb4b2a2ff27b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:29:07.614547 6487 raft_consensus.cc:482] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:29:07.614693 6487 raft_consensus.cc:504] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.615175 6487 leader_election.cc:288] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1108a5faf487493bb7703e1cae80a1e2 (127.29.78.130:42051), ac5d8094f08e4c4597eedb4b2a2ff27b (127.29.78.131:36745)
I20260808 18:29:07.615578 6383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad80bf3a8e04366b1829a0662a9ad92" candidate_uuid: "b992bf381f0a41849ffc437f30c71bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" is_pre_election: true
I20260808 18:29:07.615588 6234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad80bf3a8e04366b1829a0662a9ad92" candidate_uuid: "b992bf381f0a41849ffc437f30c71bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1108a5faf487493bb7703e1cae80a1e2" is_pre_election: true
I20260808 18:29:07.615744 6383 raft_consensus.cc:2454] T 5ad80bf3a8e04366b1829a0662a9ad92 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b992bf381f0a41849ffc437f30c71bce in term 0.
I20260808 18:29:07.615752 6234 raft_consensus.cc:2454] T 5ad80bf3a8e04366b1829a0662a9ad92 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b992bf381f0a41849ffc437f30c71bce in term 0.
I20260808 18:29:07.616041 6009 leader_election.cc:302] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1108a5faf487493bb7703e1cae80a1e2, b992bf381f0a41849ffc437f30c71bce; no voters:
I20260808 18:29:07.616235 6487 raft_consensus.cc:2781] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:29:07.616300 6487 raft_consensus.cc:482] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:29:07.616346 6487 raft_consensus.cc:3037] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:07.617470 6487 raft_consensus.cc:504] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.617833 6487 leader_election.cc:288] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce [CANDIDATE]: Term 1 election: Requested vote from peers 1108a5faf487493bb7703e1cae80a1e2 (127.29.78.130:42051), ac5d8094f08e4c4597eedb4b2a2ff27b (127.29.78.131:36745)
I20260808 18:29:07.618144 6234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad80bf3a8e04366b1829a0662a9ad92" candidate_uuid: "b992bf381f0a41849ffc437f30c71bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1108a5faf487493bb7703e1cae80a1e2"
I20260808 18:29:07.618188 6383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad80bf3a8e04366b1829a0662a9ad92" candidate_uuid: "b992bf381f0a41849ffc437f30c71bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b"
I20260808 18:29:07.618256 6234 raft_consensus.cc:3037] T 5ad80bf3a8e04366b1829a0662a9ad92 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:07.618319 6383 raft_consensus.cc:3037] T 5ad80bf3a8e04366b1829a0662a9ad92 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:07.619184 6383 raft_consensus.cc:2454] T 5ad80bf3a8e04366b1829a0662a9ad92 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b992bf381f0a41849ffc437f30c71bce in term 1.
I20260808 18:29:07.619243 6234 raft_consensus.cc:2454] T 5ad80bf3a8e04366b1829a0662a9ad92 P 1108a5faf487493bb7703e1cae80a1e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b992bf381f0a41849ffc437f30c71bce in term 1.
I20260808 18:29:07.619498 6010 leader_election.cc:302] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac5d8094f08e4c4597eedb4b2a2ff27b, b992bf381f0a41849ffc437f30c71bce; no voters:
I20260808 18:29:07.619699 6487 raft_consensus.cc:2781] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:29:07.619825 6487 raft_consensus.cc:686] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce [term 1 LEADER]: Becoming Leader. State: Replica: b992bf381f0a41849ffc437f30c71bce, State: Running, Role: LEADER
I20260808 18:29:07.619956 6487 consensus_queue.cc:237] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.620682 6479 tablet_replica.cc:439] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce: TxnStatusTablet state changed. Reason: New leader b992bf381f0a41849ffc437f30c71bce. Latest consensus state: current_term: 1 leader_uuid: "b992bf381f0a41849ffc437f30c71bce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } } }
I20260808 18:29:07.620797 6479 tablet_replica.cc:442] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce: This TxnStatusTablet replica's current role is: LEADER
I20260808 18:29:07.621119 6515 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 18:29:07.621289 6515 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 18:29:07.621675 5915 catalog_manager.cc:5495] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce reported cstate change: term changed from 0 to 1, leader changed from <none> to b992bf381f0a41849ffc437f30c71bce (127.29.78.129). New cstate: current_term: 1 leader_uuid: "b992bf381f0a41849ffc437f30c71bce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } health_report { overall_health: UNKNOWN } } }
I20260808 18:29:07.635016 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.631212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.29.78.129:39945)
0808 18:29:07.631377 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.631382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.631514 (+ 132us) server_negotiation.cc:407] Connection header received
0808 18:29:07.631793 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.631797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.631845 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.631910 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:29:07.632199 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.632810 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.633983 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.634188 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.634260 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.634607 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:29:07.634672 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:29:07.634727 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:07.634851 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 18:29:07.634899 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 18:29:07.635053 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.631085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:39945 (local address 127.0.0.1:35082)
0808 18:29:07.631432 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:07.631448 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:07.631595 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:07.631959 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:07.631963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:07.631972 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 18:29:07.632088 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.632094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.632908 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.632912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.633870 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.633881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.634287 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.634291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.634379 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.634470 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 18:29:07.634752 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 18:29:07.634765 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 18:29:07.634831 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 18:29:07.634912 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260808 18:29:07.637696 6383 raft_consensus.cc:1261] T 5ad80bf3a8e04366b1829a0662a9ad92 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 1 FOLLOWER]: Refusing update from remote peer b992bf381f0a41849ffc437f30c71bce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:29:07.637779 6234 raft_consensus.cc:1261] T 5ad80bf3a8e04366b1829a0662a9ad92 P 1108a5faf487493bb7703e1cae80a1e2 [term 1 FOLLOWER]: Refusing update from remote peer b992bf381f0a41849ffc437f30c71bce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:29:07.638175 6487 consensus_queue.cc:1035] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:29:07.638321 6479 consensus_queue.cc:1035] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:29:07.639637 6474 tablet_replica.cc:439] T 5ad80bf3a8e04366b1829a0662a9ad92 P 1108a5faf487493bb7703e1cae80a1e2: TxnStatusTablet state changed. Reason: New leader b992bf381f0a41849ffc437f30c71bce. Latest consensus state: current_term: 1 leader_uuid: "b992bf381f0a41849ffc437f30c71bce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } } }
I20260808 18:29:07.639752 6474 tablet_replica.cc:442] T 5ad80bf3a8e04366b1829a0662a9ad92 P 1108a5faf487493bb7703e1cae80a1e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:29:07.640578 6475 tablet_replica.cc:439] T 5ad80bf3a8e04366b1829a0662a9ad92 P ac5d8094f08e4c4597eedb4b2a2ff27b: TxnStatusTablet state changed. Reason: New leader b992bf381f0a41849ffc437f30c71bce. Latest consensus state: current_term: 1 leader_uuid: "b992bf381f0a41849ffc437f30c71bce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } } }
I20260808 18:29:07.640699 6475 tablet_replica.cc:442] T 5ad80bf3a8e04366b1829a0662a9ad92 P ac5d8094f08e4c4597eedb4b2a2ff27b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:29:07.641490 6476 tablet_replica.cc:439] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b992bf381f0a41849ffc437f30c71bce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } } }
I20260808 18:29:07.641610 6476 tablet_replica.cc:442] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce: This TxnStatusTablet replica's current role is: LEADER
I20260808 18:29:07.641697 6482 mvcc.cc:205] Tried to move back new op lower bound from 7316331510319755264 to 7316331510252658688. Current Snapshot: MvccSnapshot[applied={T|T < 7316331510319755264 or (T in {7316331510319755264})}]
I20260808 18:29:07.642145 6479 tablet_replica.cc:439] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b992bf381f0a41849ffc437f30c71bce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } } }
I20260808 18:29:07.642259 6479 tablet_replica.cc:442] T 5ad80bf3a8e04366b1829a0662a9ad92 P b992bf381f0a41849ffc437f30c71bce: This TxnStatusTablet replica's current role is: LEADER
I20260808 18:29:07.643908 6475 tablet_replica.cc:439] T 5ad80bf3a8e04366b1829a0662a9ad92 P ac5d8094f08e4c4597eedb4b2a2ff27b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b992bf381f0a41849ffc437f30c71bce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } } }
I20260808 18:29:07.644052 6475 tablet_replica.cc:442] T 5ad80bf3a8e04366b1829a0662a9ad92 P ac5d8094f08e4c4597eedb4b2a2ff27b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:29:07.644249 6474 tablet_replica.cc:439] T 5ad80bf3a8e04366b1829a0662a9ad92 P 1108a5faf487493bb7703e1cae80a1e2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b992bf381f0a41849ffc437f30c71bce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } } }
I20260808 18:29:07.644362 6474 tablet_replica.cc:442] T 5ad80bf3a8e04366b1829a0662a9ad92 P 1108a5faf487493bb7703e1cae80a1e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:29:07.652657 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.647945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35388 (local address 127.29.78.130:42051)
0808 18:29:07.648164 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.648168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.648181 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:29:07.648254 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.648257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.648327 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.648455 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:29:07.648774 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.649652 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.651047 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.651261 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.651352 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.651695 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:29:07.652170 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:29:07.652266 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:07.652427 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 18:29:07.652483 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260808 18:29:07.663935 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.659865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:54433 (local address 127.29.78.129:39945)
0808 18:29:07.659990 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.659993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.660262 (+ 269us) server_negotiation.cc:407] Connection header received
0808 18:29:07.660420 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.660423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.660465 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.660541 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:29:07.660821 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.661413 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.662819 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.663029 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.663113 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.663502 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:29:07.663570 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:29:07.663638 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:07.663777 (+ 139us) server_negotiation.cc:299] Negotiation successful
0808 18:29:07.663825 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 18:29:07.664022 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.659766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:39945 (local address 127.29.78.130:54433)
0808 18:29:07.660143 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:07.660159 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:07.660268 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:07.660562 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:07.660565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:07.660573 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 18:29:07.660693 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.660699 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.661533 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.661537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.662684 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.662696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.663161 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.663166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.663283 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.663339 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 18:29:07.663673 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 18:29:07.663680 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 18:29:07.663755 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 18:29:07.663848 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
I20260808 18:29:07.671459 6383 raft_consensus.cc:1261] T 0ac5b0029f8c478e95853c7cbebe53d6 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 1 FOLLOWER]: Refusing update from remote peer 1108a5faf487493bb7703e1cae80a1e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:29:07.671620 6084 raft_consensus.cc:1261] T 0ac5b0029f8c478e95853c7cbebe53d6 P b992bf381f0a41849ffc437f30c71bce [term 1 FOLLOWER]: Refusing update from remote peer 1108a5faf487493bb7703e1cae80a1e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:29:07.672015 6474 consensus_queue.cc:1035] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:29:07.672214 6502 consensus_queue.cc:1035] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:29:07.697911 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.693654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:46413 (local address 127.29.78.130:42051)
0808 18:29:07.693783 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.693787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.694049 (+ 262us) server_negotiation.cc:407] Connection header received
0808 18:29:07.694220 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.694224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.694278 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.694360 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:29:07.694680 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.695564 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.696573 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.696975 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.697083 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.697379 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:29:07.697465 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:29:07.697539 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:07.697698 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 18:29:07.697752 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 18:29:07.698369 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.693581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:42051 (local address 127.29.78.129:46413)
0808 18:29:07.693943 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:07.693958 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:07.694107 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:07.694421 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:07.694424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:07.694435 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 18:29:07.694556 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.694564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.695684 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.695687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.696450 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.696460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.697089 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.697092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.697159 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.697231 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 18:29:07.697585 (+ 354us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 18:29:07.697593 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 18:29:07.698122 (+ 529us) client_negotiation.cc:240] Negotiation successful
0808 18:29:07.698218 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":133,"threads_started":1}
I20260808 18:29:07.726227 5913 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:29:07.726444 5913 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:29:07.728740 5913 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad: Sending DeleteTablet for 3 replicas of tablet 0ac5b0029f8c478e95853c7cbebe53d6
I20260808 18:29:07.729377 6212 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ac5b0029f8c478e95853c7cbebe53d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:29:07 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:35376
I20260808 18:29:07.729523 6063 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ac5b0029f8c478e95853c7cbebe53d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:29:07 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:35060
I20260808 18:29:07.729682 6495 tablet_replica.cc:330] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2: stopping tablet replica
I20260808 18:29:07.729728 6494 tablet_replica.cc:330] T 0ac5b0029f8c478e95853c7cbebe53d6 P b992bf381f0a41849ffc437f30c71bce: stopping tablet replica
I20260808 18:29:07.729821 6495 raft_consensus.cc:2229] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:29:07.729846 6494 raft_consensus.cc:2229] T 0ac5b0029f8c478e95853c7cbebe53d6 P b992bf381f0a41849ffc437f30c71bce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:29:07.729972 6494 raft_consensus.cc:2258] T 0ac5b0029f8c478e95853c7cbebe53d6 P b992bf381f0a41849ffc437f30c71bce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:29:07.730093 6495 raft_consensus.cc:2258] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:29:07.730262 6361 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ac5b0029f8c478e95853c7cbebe53d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:29:07 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:56150
I20260808 18:29:07.730561 6492 tablet_replica.cc:330] T 0ac5b0029f8c478e95853c7cbebe53d6 P ac5d8094f08e4c4597eedb4b2a2ff27b: stopping tablet replica
I20260808 18:29:07.730669 6492 raft_consensus.cc:2229] T 0ac5b0029f8c478e95853c7cbebe53d6 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:29:07.730767 6492 raft_consensus.cc:2258] T 0ac5b0029f8c478e95853c7cbebe53d6 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:29:07.731065 6494 ts_tablet_manager.cc:1905] T 0ac5b0029f8c478e95853c7cbebe53d6 P b992bf381f0a41849ffc437f30c71bce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:29:07.731652 6492 ts_tablet_manager.cc:1905] T 0ac5b0029f8c478e95853c7cbebe53d6 P ac5d8094f08e4c4597eedb4b2a2ff27b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:29:07.734264 6492 ts_tablet_manager.cc:1918] T 0ac5b0029f8c478e95853c7cbebe53d6 P ac5d8094f08e4c4597eedb4b2a2ff27b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:29:07.734328 6492 log.cc:1196] T 0ac5b0029f8c478e95853c7cbebe53d6 P ac5d8094f08e4c4597eedb4b2a2ff27b: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/0ac5b0029f8c478e95853c7cbebe53d6
I20260808 18:29:07.734303 6494 ts_tablet_manager.cc:1918] T 0ac5b0029f8c478e95853c7cbebe53d6 P b992bf381f0a41849ffc437f30c71bce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:29:07.734381 6494 log.cc:1196] T 0ac5b0029f8c478e95853c7cbebe53d6 P b992bf381f0a41849ffc437f30c71bce: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-0/wal/wals/0ac5b0029f8c478e95853c7cbebe53d6
I20260808 18:29:07.735008 6495 ts_tablet_manager.cc:1905] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:29:07.735055 6494 ts_tablet_manager.cc:1939] T 0ac5b0029f8c478e95853c7cbebe53d6 P b992bf381f0a41849ffc437f30c71bce: Deleting consensus metadata
I20260808 18:29:07.737612 6495 ts_tablet_manager.cc:1918] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:29:07.737864 5900 catalog_manager.cc:4841] TS b992bf381f0a41849ffc437f30c71bce (127.29.78.129:39945): tablet 0ac5b0029f8c478e95853c7cbebe53d6 (table test-table [id=2bb430669e354a02a113c656abac9f24]) successfully deleted
I20260808 18:29:07.738653 6492 ts_tablet_manager.cc:1939] T 0ac5b0029f8c478e95853c7cbebe53d6 P ac5d8094f08e4c4597eedb4b2a2ff27b: Deleting consensus metadata
I20260808 18:29:07.738859 6495 log.cc:1196] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/0ac5b0029f8c478e95853c7cbebe53d6
I20260808 18:29:07.739127 6495 ts_tablet_manager.cc:1939] T 0ac5b0029f8c478e95853c7cbebe53d6 P 1108a5faf487493bb7703e1cae80a1e2: Deleting consensus metadata
I20260808 18:29:07.739727 5899 catalog_manager.cc:4841] TS 1108a5faf487493bb7703e1cae80a1e2 (127.29.78.130:42051): tablet 0ac5b0029f8c478e95853c7cbebe53d6 (table test-table [id=2bb430669e354a02a113c656abac9f24]) successfully deleted
I20260808 18:29:07.740460 5900 catalog_manager.cc:4841] TS ac5d8094f08e4c4597eedb4b2a2ff27b (127.29.78.131:36745): tablet 0ac5b0029f8c478e95853c7cbebe53d6 (table test-table [id=2bb430669e354a02a113c656abac9f24]) successfully deleted
Aug 08 18:29:07 dist-test-slave-gp6n krb5kdc[5866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 18:29:07 dist-test-slave-gp6n krb5kdc[5866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:07 dist-test-slave-gp6n krb5kdc[5866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
W20260808 18:29:07.775622 5966 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:07.767169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41836 (local address 127.29.78.190:42985)
0808 18:29:07.767305 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.767312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.767531 (+ 219us) server_negotiation.cc:407] Connection header received
0808 18:29:07.767674 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.767678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.767741 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.767842 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:07.768795 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.769735 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.771069 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.771461 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.771581 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.775476 (+ 3895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41836: BlockingRecv error: recv got EOF from 127.0.0.1:41836 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 08 18:29:07 dist-test-slave-gp6n krb5kdc[5866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260808 18:29:07.788734 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.777720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41848 (local address 127.29.78.190:42985)
0808 18:29:07.777831 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.777835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.778303 (+ 468us) server_negotiation.cc:407] Connection header received
0808 18:29:07.778455 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.778458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.778509 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.778599 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:07.779357 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.780223 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.781324 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.781697 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.781809 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.784766 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:07.784771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:07.784774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:07.784804 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:07.786890 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:07.787429 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:07.787433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:07.787435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:07.787528 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:07.787843 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:07.787846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:07.787848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:07.788069 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:07.788196 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:07.788460 (+ 264us) server_negotiation.cc:299] Negotiation successful
0808 18:29:07.788573 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 18:29:07.797197 5913 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:29:07.797663 5913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:29:07.803505 6063 tablet_service.cc:1467] Processing CreateTablet for tablet 2f40255cbc1c4a4582d450c4ae57f4e3 (DEFAULT_TABLE table=test-table [id=9c32d08424fc46f38f8e51de88567779]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:29:07.803834 6063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f40255cbc1c4a4582d450c4ae57f4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:07.806521 6468 tablet_bootstrap.cc:492] T 2f40255cbc1c4a4582d450c4ae57f4e3 P b992bf381f0a41849ffc437f30c71bce: Bootstrap starting.
I20260808 18:29:07.807426 6468 tablet_bootstrap.cc:654] T 2f40255cbc1c4a4582d450c4ae57f4e3 P b992bf381f0a41849ffc437f30c71bce: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:07.808323 6212 tablet_service.cc:1467] Processing CreateTablet for tablet 2f40255cbc1c4a4582d450c4ae57f4e3 (DEFAULT_TABLE table=test-table [id=9c32d08424fc46f38f8e51de88567779]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:29:07.808574 6212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f40255cbc1c4a4582d450c4ae57f4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:07.810714 6467 tablet_bootstrap.cc:492] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2: Bootstrap starting.
I20260808 18:29:07.810858 6361 tablet_service.cc:1467] Processing CreateTablet for tablet 2f40255cbc1c4a4582d450c4ae57f4e3 (DEFAULT_TABLE table=test-table [id=9c32d08424fc46f38f8e51de88567779]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:29:07.811162 6361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f40255cbc1c4a4582d450c4ae57f4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:07.811666 6467 tablet_bootstrap.cc:654] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:07.813292 6469 tablet_bootstrap.cc:492] T 2f40255cbc1c4a4582d450c4ae57f4e3 P ac5d8094f08e4c4597eedb4b2a2ff27b: Bootstrap starting.
I20260808 18:29:07.814193 6469 tablet_bootstrap.cc:654] T 2f40255cbc1c4a4582d450c4ae57f4e3 P ac5d8094f08e4c4597eedb4b2a2ff27b: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:07.816061 6468 tablet_bootstrap.cc:492] T 2f40255cbc1c4a4582d450c4ae57f4e3 P b992bf381f0a41849ffc437f30c71bce: No bootstrap required, opened a new log
I20260808 18:29:07.816162 6468 ts_tablet_manager.cc:1397] T 2f40255cbc1c4a4582d450c4ae57f4e3 P b992bf381f0a41849ffc437f30c71bce: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260808 18:29:07.816334 6467 tablet_bootstrap.cc:492] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2: No bootstrap required, opened a new log
I20260808 18:29:07.816407 6467 ts_tablet_manager.cc:1397] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260808 18:29:07.816606 6468 raft_consensus.cc:348] T 2f40255cbc1c4a4582d450c4ae57f4e3 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.816716 6468 raft_consensus.cc:374] T 2f40255cbc1c4a4582d450c4ae57f4e3 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:07.816751 6468 raft_consensus.cc:729] T 2f40255cbc1c4a4582d450c4ae57f4e3 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b992bf381f0a41849ffc437f30c71bce, State: Initialized, Role: FOLLOWER
I20260808 18:29:07.816849 6467 raft_consensus.cc:348] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.816881 6468 consensus_queue.cc:260] T 2f40255cbc1c4a4582d450c4ae57f4e3 P b992bf381f0a41849ffc437f30c71bce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.816957 6467 raft_consensus.cc:374] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:07.816992 6467 raft_consensus.cc:729] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1108a5faf487493bb7703e1cae80a1e2, State: Initialized, Role: FOLLOWER
I20260808 18:29:07.817116 6467 consensus_queue.cc:260] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.817178 6468 ts_tablet_manager.cc:1428] T 2f40255cbc1c4a4582d450c4ae57f4e3 P b992bf381f0a41849ffc437f30c71bce: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 18:29:07.817371 6467 ts_tablet_manager.cc:1428] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:29:07.823115 6469 tablet_bootstrap.cc:492] T 2f40255cbc1c4a4582d450c4ae57f4e3 P ac5d8094f08e4c4597eedb4b2a2ff27b: No bootstrap required, opened a new log
I20260808 18:29:07.823240 6469 ts_tablet_manager.cc:1397] T 2f40255cbc1c4a4582d450c4ae57f4e3 P ac5d8094f08e4c4597eedb4b2a2ff27b: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260808 18:29:07.823730 6469 raft_consensus.cc:348] T 2f40255cbc1c4a4582d450c4ae57f4e3 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.823882 6469 raft_consensus.cc:374] T 2f40255cbc1c4a4582d450c4ae57f4e3 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:07.823926 6469 raft_consensus.cc:729] T 2f40255cbc1c4a4582d450c4ae57f4e3 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac5d8094f08e4c4597eedb4b2a2ff27b, State: Initialized, Role: FOLLOWER
I20260808 18:29:07.824072 6469 consensus_queue.cc:260] T 2f40255cbc1c4a4582d450c4ae57f4e3 P ac5d8094f08e4c4597eedb4b2a2ff27b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.824348 6469 ts_tablet_manager.cc:1428] T 2f40255cbc1c4a4582d450c4ae57f4e3 P ac5d8094f08e4c4597eedb4b2a2ff27b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:29:07.836069 6525 raft_consensus.cc:482] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:29:07.836189 6525 raft_consensus.cc:504] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.836597 6525 leader_election.cc:288] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b992bf381f0a41849ffc437f30c71bce (127.29.78.129:39945), ac5d8094f08e4c4597eedb4b2a2ff27b (127.29.78.131:36745)
I20260808 18:29:07.836905 6084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f40255cbc1c4a4582d450c4ae57f4e3" candidate_uuid: "1108a5faf487493bb7703e1cae80a1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b992bf381f0a41849ffc437f30c71bce" is_pre_election: true
I20260808 18:29:07.836979 6383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f40255cbc1c4a4582d450c4ae57f4e3" candidate_uuid: "1108a5faf487493bb7703e1cae80a1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" is_pre_election: true
I20260808 18:29:07.837076 6084 raft_consensus.cc:2454] T 2f40255cbc1c4a4582d450c4ae57f4e3 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1108a5faf487493bb7703e1cae80a1e2 in term 0.
I20260808 18:29:07.837113 6383 raft_consensus.cc:2454] T 2f40255cbc1c4a4582d450c4ae57f4e3 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1108a5faf487493bb7703e1cae80a1e2 in term 0.
I20260808 18:29:07.837369 6162 leader_election.cc:302] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1108a5faf487493bb7703e1cae80a1e2, b992bf381f0a41849ffc437f30c71bce; no voters:
I20260808 18:29:07.837678 6525 raft_consensus.cc:2781] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:29:07.837744 6525 raft_consensus.cc:482] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:29:07.837787 6525 raft_consensus.cc:3037] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:07.839102 6525 raft_consensus.cc:504] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.839488 6525 leader_election.cc:288] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2 [CANDIDATE]: Term 1 election: Requested vote from peers b992bf381f0a41849ffc437f30c71bce (127.29.78.129:39945), ac5d8094f08e4c4597eedb4b2a2ff27b (127.29.78.131:36745)
I20260808 18:29:07.839784 6084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f40255cbc1c4a4582d450c4ae57f4e3" candidate_uuid: "1108a5faf487493bb7703e1cae80a1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b992bf381f0a41849ffc437f30c71bce"
I20260808 18:29:07.839860 6383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f40255cbc1c4a4582d450c4ae57f4e3" candidate_uuid: "1108a5faf487493bb7703e1cae80a1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b"
I20260808 18:29:07.839906 6084 raft_consensus.cc:3037] T 2f40255cbc1c4a4582d450c4ae57f4e3 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:07.839960 6383 raft_consensus.cc:3037] T 2f40255cbc1c4a4582d450c4ae57f4e3 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:07.841080 6383 raft_consensus.cc:2454] T 2f40255cbc1c4a4582d450c4ae57f4e3 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1108a5faf487493bb7703e1cae80a1e2 in term 1.
I20260808 18:29:07.841085 6084 raft_consensus.cc:2454] T 2f40255cbc1c4a4582d450c4ae57f4e3 P b992bf381f0a41849ffc437f30c71bce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1108a5faf487493bb7703e1cae80a1e2 in term 1.
I20260808 18:29:07.841367 6162 leader_election.cc:302] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1108a5faf487493bb7703e1cae80a1e2, b992bf381f0a41849ffc437f30c71bce; no voters:
I20260808 18:29:07.841740 6525 raft_consensus.cc:2781] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:29:07.841835 6525 raft_consensus.cc:686] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2 [term 1 LEADER]: Becoming Leader. State: Replica: 1108a5faf487493bb7703e1cae80a1e2, State: Running, Role: LEADER
I20260808 18:29:07.841970 6525 consensus_queue.cc:237] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.843756 5913 catalog_manager.cc:5495] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1108a5faf487493bb7703e1cae80a1e2 (127.29.78.130). New cstate: current_term: 1 leader_uuid: "1108a5faf487493bb7703e1cae80a1e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } health_report { overall_health: UNKNOWN } } }
I20260808 18:29:07.855506 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.852362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35390 (local address 127.29.78.130:42051)
0808 18:29:07.852471 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.852477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.852556 (+ 79us) server_negotiation.cc:407] Connection header received
0808 18:29:07.852714 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.852717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.852769 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.852849 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:29:07.853093 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.853590 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.854480 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.854700 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.854805 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.855081 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:29:07.855149 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:29:07.855231 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:07.855320 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 18:29:07.855368 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:29:07.857548 6084 raft_consensus.cc:1261] T 2f40255cbc1c4a4582d450c4ae57f4e3 P b992bf381f0a41849ffc437f30c71bce [term 1 FOLLOWER]: Refusing update from remote peer 1108a5faf487493bb7703e1cae80a1e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:29:07.857584 6383 raft_consensus.cc:1261] T 2f40255cbc1c4a4582d450c4ae57f4e3 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 1 FOLLOWER]: Refusing update from remote peer 1108a5faf487493bb7703e1cae80a1e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:29:07.858040 6502 consensus_queue.cc:1035] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:29:07.858181 6525 consensus_queue.cc:1035] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:29:07.861718 6483 mvcc.cc:205] Tried to move back new op lower bound from 7316331511220002816 to 7316331511162183680. Current Snapshot: MvccSnapshot[applied={T|T < 7316331511220002816}]
I20260808 18:29:07.866600 5913 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:29:07.866761 5913 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:29:07.868880 5913 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad: Sending DeleteTablet for 3 replicas of tablet 2f40255cbc1c4a4582d450c4ae57f4e3
I20260808 18:29:07.869418 6063 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f40255cbc1c4a4582d450c4ae57f4e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:29:07 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:35060
I20260808 18:29:07.869644 6494 tablet_replica.cc:330] T 2f40255cbc1c4a4582d450c4ae57f4e3 P b992bf381f0a41849ffc437f30c71bce: stopping tablet replica
I20260808 18:29:07.869700 6361 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f40255cbc1c4a4582d450c4ae57f4e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:29:07 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:56150
I20260808 18:29:07.869789 6494 raft_consensus.cc:2229] T 2f40255cbc1c4a4582d450c4ae57f4e3 P b992bf381f0a41849ffc437f30c71bce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:29:07.869870 6492 tablet_replica.cc:330] T 2f40255cbc1c4a4582d450c4ae57f4e3 P ac5d8094f08e4c4597eedb4b2a2ff27b: stopping tablet replica
I20260808 18:29:07.869927 6494 raft_consensus.cc:2258] T 2f40255cbc1c4a4582d450c4ae57f4e3 P b992bf381f0a41849ffc437f30c71bce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:29:07.869994 6492 raft_consensus.cc:2229] T 2f40255cbc1c4a4582d450c4ae57f4e3 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:29:07.870133 6492 raft_consensus.cc:2258] T 2f40255cbc1c4a4582d450c4ae57f4e3 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:29:07.870149 6212 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f40255cbc1c4a4582d450c4ae57f4e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:29:07 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:35376
I20260808 18:29:07.870297 6495 tablet_replica.cc:330] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2: stopping tablet replica
I20260808 18:29:07.870440 6495 raft_consensus.cc:2229] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:29:07.870621 6495 raft_consensus.cc:2258] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:29:07.870761 5913 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:29:07.871201 5913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:29:07.871441 6495 ts_tablet_manager.cc:1905] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:29:07.872315 6494 ts_tablet_manager.cc:1905] T 2f40255cbc1c4a4582d450c4ae57f4e3 P b992bf381f0a41849ffc437f30c71bce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:29:07.874053 6495 ts_tablet_manager.cc:1918] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:29:07.874119 6495 log.cc:1196] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/2f40255cbc1c4a4582d450c4ae57f4e3
I20260808 18:29:07.874377 6495 ts_tablet_manager.cc:1939] T 2f40255cbc1c4a4582d450c4ae57f4e3 P 1108a5faf487493bb7703e1cae80a1e2: Deleting consensus metadata
I20260808 18:29:07.874982 6494 ts_tablet_manager.cc:1918] T 2f40255cbc1c4a4582d450c4ae57f4e3 P b992bf381f0a41849ffc437f30c71bce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:29:07.875077 6494 log.cc:1196] T 2f40255cbc1c4a4582d450c4ae57f4e3 P b992bf381f0a41849ffc437f30c71bce: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-0/wal/wals/2f40255cbc1c4a4582d450c4ae57f4e3
I20260808 18:29:07.875236 5899 catalog_manager.cc:4841] TS 1108a5faf487493bb7703e1cae80a1e2 (127.29.78.130:42051): tablet 2f40255cbc1c4a4582d450c4ae57f4e3 (table test-table [id=9c32d08424fc46f38f8e51de88567779]) successfully deleted
I20260808 18:29:07.875347 6494 ts_tablet_manager.cc:1939] T 2f40255cbc1c4a4582d450c4ae57f4e3 P b992bf381f0a41849ffc437f30c71bce: Deleting consensus metadata
I20260808 18:29:07.876000 5900 catalog_manager.cc:4841] TS b992bf381f0a41849ffc437f30c71bce (127.29.78.129:39945): tablet 2f40255cbc1c4a4582d450c4ae57f4e3 (table test-table [id=9c32d08424fc46f38f8e51de88567779]) successfully deleted
I20260808 18:29:07.876454 6492 ts_tablet_manager.cc:1905] T 2f40255cbc1c4a4582d450c4ae57f4e3 P ac5d8094f08e4c4597eedb4b2a2ff27b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:29:07.878906 6492 ts_tablet_manager.cc:1918] T 2f40255cbc1c4a4582d450c4ae57f4e3 P ac5d8094f08e4c4597eedb4b2a2ff27b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:29:07.878988 6492 log.cc:1196] T 2f40255cbc1c4a4582d450c4ae57f4e3 P ac5d8094f08e4c4597eedb4b2a2ff27b: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/2f40255cbc1c4a4582d450c4ae57f4e3
I20260808 18:29:07.878980 6063 tablet_service.cc:1467] Processing CreateTablet for tablet d21325e226b743458e030090b64776f7 (DEFAULT_TABLE table=test-table [id=c9524e85002a4fe99144bcf3a9379d95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:29:07.879256 6063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d21325e226b743458e030090b64776f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:07.879170 6361 tablet_service.cc:1467] Processing CreateTablet for tablet d21325e226b743458e030090b64776f7 (DEFAULT_TABLE table=test-table [id=c9524e85002a4fe99144bcf3a9379d95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:29:07.879253 6492 ts_tablet_manager.cc:1939] T 2f40255cbc1c4a4582d450c4ae57f4e3 P ac5d8094f08e4c4597eedb4b2a2ff27b: Deleting consensus metadata
I20260808 18:29:07.879491 6361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d21325e226b743458e030090b64776f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:07.879937 5900 catalog_manager.cc:4841] TS ac5d8094f08e4c4597eedb4b2a2ff27b (127.29.78.131:36745): tablet 2f40255cbc1c4a4582d450c4ae57f4e3 (table test-table [id=9c32d08424fc46f38f8e51de88567779]) successfully deleted
I20260808 18:29:07.881335 6468 tablet_bootstrap.cc:492] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce: Bootstrap starting.
I20260808 18:29:07.881850 6469 tablet_bootstrap.cc:492] T d21325e226b743458e030090b64776f7 P ac5d8094f08e4c4597eedb4b2a2ff27b: Bootstrap starting.
I20260808 18:29:07.882202 6468 tablet_bootstrap.cc:654] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:07.882689 6212 tablet_service.cc:1467] Processing CreateTablet for tablet d21325e226b743458e030090b64776f7 (DEFAULT_TABLE table=test-table [id=c9524e85002a4fe99144bcf3a9379d95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:29:07.882817 6469 tablet_bootstrap.cc:654] T d21325e226b743458e030090b64776f7 P ac5d8094f08e4c4597eedb4b2a2ff27b: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:07.882962 6212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d21325e226b743458e030090b64776f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:07.884127 6469 tablet_bootstrap.cc:492] T d21325e226b743458e030090b64776f7 P ac5d8094f08e4c4597eedb4b2a2ff27b: No bootstrap required, opened a new log
I20260808 18:29:07.884217 6469 ts_tablet_manager.cc:1397] T d21325e226b743458e030090b64776f7 P ac5d8094f08e4c4597eedb4b2a2ff27b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 18:29:07.884697 6469 raft_consensus.cc:348] T d21325e226b743458e030090b64776f7 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.884810 6469 raft_consensus.cc:374] T d21325e226b743458e030090b64776f7 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:07.884852 6469 raft_consensus.cc:729] T d21325e226b743458e030090b64776f7 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac5d8094f08e4c4597eedb4b2a2ff27b, State: Initialized, Role: FOLLOWER
I20260808 18:29:07.884958 6469 consensus_queue.cc:260] T d21325e226b743458e030090b64776f7 P ac5d8094f08e4c4597eedb4b2a2ff27b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.885290 6467 tablet_bootstrap.cc:492] T d21325e226b743458e030090b64776f7 P 1108a5faf487493bb7703e1cae80a1e2: Bootstrap starting.
I20260808 18:29:07.885407 6469 ts_tablet_manager.cc:1428] T d21325e226b743458e030090b64776f7 P ac5d8094f08e4c4597eedb4b2a2ff27b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260808 18:29:07.886106 6467 tablet_bootstrap.cc:654] T d21325e226b743458e030090b64776f7 P 1108a5faf487493bb7703e1cae80a1e2: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:07.887993 6467 tablet_bootstrap.cc:492] T d21325e226b743458e030090b64776f7 P 1108a5faf487493bb7703e1cae80a1e2: No bootstrap required, opened a new log
I20260808 18:29:07.888077 6467 ts_tablet_manager.cc:1397] T d21325e226b743458e030090b64776f7 P 1108a5faf487493bb7703e1cae80a1e2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 18:29:07.888449 6467 raft_consensus.cc:348] T d21325e226b743458e030090b64776f7 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.888548 6467 raft_consensus.cc:374] T d21325e226b743458e030090b64776f7 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:07.888571 6467 raft_consensus.cc:729] T d21325e226b743458e030090b64776f7 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1108a5faf487493bb7703e1cae80a1e2, State: Initialized, Role: FOLLOWER
I20260808 18:29:07.888646 6467 consensus_queue.cc:260] T d21325e226b743458e030090b64776f7 P 1108a5faf487493bb7703e1cae80a1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.888898 6467 ts_tablet_manager.cc:1428] T d21325e226b743458e030090b64776f7 P 1108a5faf487493bb7703e1cae80a1e2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:29:07.889566 6468 tablet_bootstrap.cc:492] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce: No bootstrap required, opened a new log
I20260808 18:29:07.889639 6468 ts_tablet_manager.cc:1397] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260808 18:29:07.890010 6468 raft_consensus.cc:348] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.890106 6468 raft_consensus.cc:374] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:07.890128 6468 raft_consensus.cc:729] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b992bf381f0a41849ffc437f30c71bce, State: Initialized, Role: FOLLOWER
I20260808 18:29:07.890249 6468 consensus_queue.cc:260] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.890549 6468 ts_tablet_manager.cc:1428] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:29:07.916754 6487 raft_consensus.cc:482] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:29:07.916884 6487 raft_consensus.cc:504] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.917346 6487 leader_election.cc:288] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1108a5faf487493bb7703e1cae80a1e2 (127.29.78.130:42051), ac5d8094f08e4c4597eedb4b2a2ff27b (127.29.78.131:36745)
I20260808 18:29:07.917747 6234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d21325e226b743458e030090b64776f7" candidate_uuid: "b992bf381f0a41849ffc437f30c71bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1108a5faf487493bb7703e1cae80a1e2" is_pre_election: true
I20260808 18:29:07.917793 6383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d21325e226b743458e030090b64776f7" candidate_uuid: "b992bf381f0a41849ffc437f30c71bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" is_pre_election: true
I20260808 18:29:07.917913 6234 raft_consensus.cc:2454] T d21325e226b743458e030090b64776f7 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b992bf381f0a41849ffc437f30c71bce in term 0.
I20260808 18:29:07.917984 6383 raft_consensus.cc:2454] T d21325e226b743458e030090b64776f7 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b992bf381f0a41849ffc437f30c71bce in term 0.
I20260808 18:29:07.918290 6010 leader_election.cc:302] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1108a5faf487493bb7703e1cae80a1e2, ac5d8094f08e4c4597eedb4b2a2ff27b, b992bf381f0a41849ffc437f30c71bce; no voters:
I20260808 18:29:07.918475 6487 raft_consensus.cc:2781] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:29:07.918546 6487 raft_consensus.cc:482] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:29:07.918598 6487 raft_consensus.cc:3037] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:07.919611 6487 raft_consensus.cc:504] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.919957 6487 leader_election.cc:288] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce [CANDIDATE]: Term 1 election: Requested vote from peers 1108a5faf487493bb7703e1cae80a1e2 (127.29.78.130:42051), ac5d8094f08e4c4597eedb4b2a2ff27b (127.29.78.131:36745)
I20260808 18:29:07.920305 6383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d21325e226b743458e030090b64776f7" candidate_uuid: "b992bf381f0a41849ffc437f30c71bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b"
I20260808 18:29:07.920305 6234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d21325e226b743458e030090b64776f7" candidate_uuid: "b992bf381f0a41849ffc437f30c71bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1108a5faf487493bb7703e1cae80a1e2"
I20260808 18:29:07.920442 6383 raft_consensus.cc:3037] T d21325e226b743458e030090b64776f7 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:07.920439 6234 raft_consensus.cc:3037] T d21325e226b743458e030090b64776f7 P 1108a5faf487493bb7703e1cae80a1e2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:07.921397 6383 raft_consensus.cc:2454] T d21325e226b743458e030090b64776f7 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b992bf381f0a41849ffc437f30c71bce in term 1.
I20260808 18:29:07.921725 6234 raft_consensus.cc:2454] T d21325e226b743458e030090b64776f7 P 1108a5faf487493bb7703e1cae80a1e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b992bf381f0a41849ffc437f30c71bce in term 1.
I20260808 18:29:07.921746 6010 leader_election.cc:302] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac5d8094f08e4c4597eedb4b2a2ff27b, b992bf381f0a41849ffc437f30c71bce; no voters:
I20260808 18:29:07.921914 6487 raft_consensus.cc:2781] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:29:07.921994 6487 raft_consensus.cc:686] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce [term 1 LEADER]: Becoming Leader. State: Replica: b992bf381f0a41849ffc437f30c71bce, State: Running, Role: LEADER
I20260808 18:29:07.922132 6487 consensus_queue.cc:237] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } }
I20260808 18:29:07.924127 5914 catalog_manager.cc:5495] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce reported cstate change: term changed from 0 to 1, leader changed from <none> to b992bf381f0a41849ffc437f30c71bce (127.29.78.129). New cstate: current_term: 1 leader_uuid: "b992bf381f0a41849ffc437f30c71bce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b992bf381f0a41849ffc437f30c71bce" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 } health_report { overall_health: UNKNOWN } } }
I20260808 18:29:07.945032 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.942022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.29.78.129:39945)
0808 18:29:07.942161 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.942165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.942178 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:29:07.942218 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.942220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.942263 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.942334 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:29:07.942621 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.943253 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.944144 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.944369 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.944448 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.944665 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:29:07.944734 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:29:07.944794 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:07.944875 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 18:29:07.944921 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 18:29:07.953141 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.950473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:46497 (local address 127.29.78.129:39945)
0808 18:29:07.950603 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:07.950606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:07.950619 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:29:07.950654 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:07.950657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:07.950697 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:07.950764 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:29:07.950961 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.951485 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.952260 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.952480 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.952579 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.952794 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:29:07.952854 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:29:07.952910 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:07.952988 (+ 78us) server_negotiation.cc:299] Negotiation successful
0808 18:29:07.953033 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":49}
I20260808 18:29:07.953147 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:07.950202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:39945 (local address 127.29.78.129:46497)
0808 18:29:07.950337 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:07.950349 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:07.950449 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:07.950784 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:07.950786 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:07.950792 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 18:29:07.950865 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.950869 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.951577 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.951580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.952133 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:07.952141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:07.952585 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:07.952587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:07.952668 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:07.952691 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 18:29:07.952928 (+ 237us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 18:29:07.952932 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 18:29:07.952980 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 18:29:07.953048 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":15}
I20260808 18:29:07.958776 6383 raft_consensus.cc:1261] T d21325e226b743458e030090b64776f7 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 1 FOLLOWER]: Refusing update from remote peer b992bf381f0a41849ffc437f30c71bce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:29:07.958791 6234 raft_consensus.cc:1261] T d21325e226b743458e030090b64776f7 P 1108a5faf487493bb7703e1cae80a1e2 [term 1 FOLLOWER]: Refusing update from remote peer b992bf381f0a41849ffc437f30c71bce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:29:07.959182 6476 consensus_queue.cc:1035] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1108a5faf487493bb7703e1cae80a1e2" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:29:07.959452 6487 consensus_queue.cc:1035] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac5d8094f08e4c4597eedb4b2a2ff27b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:29:07.991356 5913 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:29:07.991516 5913 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:29:07.993651 5913 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c651e634ac5249109a0abd50e8d32fad: Sending DeleteTablet for 3 replicas of tablet d21325e226b743458e030090b64776f7
I20260808 18:29:07.994228 6063 tablet_service.cc:1514] Processing DeleteTablet for tablet d21325e226b743458e030090b64776f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:29:07 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:35060
I20260808 18:29:07.994333 6212 tablet_service.cc:1514] Processing DeleteTablet for tablet d21325e226b743458e030090b64776f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:29:07 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:35376
I20260808 18:29:07.994333 6361 tablet_service.cc:1514] Processing DeleteTablet for tablet d21325e226b743458e030090b64776f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:29:07 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:56150
I20260808 18:29:07.994495 6494 tablet_replica.cc:330] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce: stopping tablet replica
I20260808 18:29:07.994572 6492 tablet_replica.cc:330] T d21325e226b743458e030090b64776f7 P ac5d8094f08e4c4597eedb4b2a2ff27b: stopping tablet replica
I20260808 18:29:07.994655 6494 raft_consensus.cc:2229] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:29:07.994691 6492 raft_consensus.cc:2229] T d21325e226b743458e030090b64776f7 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:29:07.994812 6492 raft_consensus.cc:2258] T d21325e226b743458e030090b64776f7 P ac5d8094f08e4c4597eedb4b2a2ff27b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:29:07.994855 6494 raft_consensus.cc:2258] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:29:07.995013 6495 tablet_replica.cc:330] T d21325e226b743458e030090b64776f7 P 1108a5faf487493bb7703e1cae80a1e2: stopping tablet replica
I20260808 18:29:07.995162 6495 raft_consensus.cc:2229] T d21325e226b743458e030090b64776f7 P 1108a5faf487493bb7703e1cae80a1e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:29:07.995288 6495 raft_consensus.cc:2258] T d21325e226b743458e030090b64776f7 P 1108a5faf487493bb7703e1cae80a1e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:29:07.995571 6492 ts_tablet_manager.cc:1905] T d21325e226b743458e030090b64776f7 P ac5d8094f08e4c4597eedb4b2a2ff27b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:29:07.996048 6494 ts_tablet_manager.cc:1905] T d21325e226b743458e030090b64776f7 P b992bf381f0a41849ffc437f30c71bce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:29:07.996061 6495 ts_tablet_manager.cc:1905] T d21325e226b743458e030090b64776f7 P 1108a5faf487493bb7703e1cae80a1e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:29:07.998188 6492 ts_tablet_manager.cc:1918] T d21325e226b743458e030090b64776f7 P ac5d8094f08e4c4597eedb4b2a2ff27b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:29:07.998281 6492 log.cc:1196] T d21325e226b743458e030090b64776f7 P ac5d8094f08e4c4597eedb4b2a2ff27b: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/d21325e226b743458e030090b64776f7
I20260808 18:29:07.998610 6492 ts_tablet_manager.cc:1939] T d21325e226b743458e030090b64776f7 P ac5d8094f08e4c4597eedb4b2a2ff27b: Deleting consensus metadata
I20260808 18:29:07.998613 6495 ts_tablet_manager.cc:1918] T d21325e226b743458e030090b64776f7 P 1108a5faf487493bb7703e1cae80a1e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:29:07.998677 6495 log.cc:1196] T d21325e226b743458e030090b64776f7 P 1108a5faf487493bb7703e1cae80a1e2: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/d21325e226b743458e030090b64776f7
I20260808 18:29:07.998714 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 5991
I20260808 18:29:08.000612 5900 catalog_manager.cc:4841] TS ac5d8094f08e4c4597eedb4b2a2ff27b (127.29.78.131:36745): tablet d21325e226b743458e030090b64776f7 (table test-table [id=c9524e85002a4fe99144bcf3a9379d95]) successfully deleted
I20260808 18:29:08.003491 6495 ts_tablet_manager.cc:1939] T d21325e226b743458e030090b64776f7 P 1108a5faf487493bb7703e1cae80a1e2: Deleting consensus metadata
I20260808 18:29:08.004148 5899 catalog_manager.cc:4841] TS 1108a5faf487493bb7703e1cae80a1e2 (127.29.78.130:42051): tablet d21325e226b743458e030090b64776f7 (table test-table [id=c9524e85002a4fe99144bcf3a9379d95]) successfully deleted
W20260808 18:29:08.011270 5900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:39945 (error 108)
I20260808 18:29:08.012332 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:08.011979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:39945 (local address 127.0.0.1:35100)
0808 18:29:08.012186 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:08.012248 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":60,"threads_started":1}
W20260808 18:29:08.012581 5900 catalog_manager.cc:4584] TS b992bf381f0a41849ffc437f30c71bce (127.29.78.129:39945): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d21325e226b743458e030090b64776f7: Network error: Client connection negotiation failed: client connection to 127.29.78.129:39945: connect: Connection refused (error 111)
I20260808 18:29:08.012899 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 6143
I20260808 18:29:08.022186 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 6290
I20260808 18:29:08.029349 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 5880
2026-08-08T18:29:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2952 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:29:08 dist-test-slave-gp6n krb5kdc[6563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:29:08 dist-test-slave-gp6n krb5kdc[6563](info): set up 2 sockets
Aug 08 18:29:08 dist-test-slave-gp6n krb5kdc[6563](info): commencing operation
krb5kdc: starting...
W20260808 18:29:09.152354 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:29:09 dist-test-slave-gp6n krb5kdc[6563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:29:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:29:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:29:09.281951 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:41367
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:33049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:41367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:29:09.395782 6577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:09.396093 6577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:09.396186 6577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:09.400261 6577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:29:09.400363 6577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:09.400405 6577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:09.400466 6577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:29:09.400508 6577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:29:09.405858 6577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:33049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:41367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:41367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:09.407064 6577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:09.408183 6577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:09.417039 6583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:09.417053 6585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:09.417044 6582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:09.417306 6577 server_base.cc:1034] running on GCE node
I20260808 18:29:09.417877 6577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:09.419023 6577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:09.420186 6577 hybrid_clock.cc:648] HybridClock initialized: now 1786213749420168 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-08T18:29:09Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1398 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:29:09 dist-test-slave-gp6n krb5kdc[6590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:29:09 dist-test-slave-gp6n krb5kdc[6590](info): set up 2 sockets
Aug 08 18:29:09 dist-test-slave-gp6n krb5kdc[6590](info): commencing operation
krb5kdc: starting...
W20260808 18:29:10.573587 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:29:10 dist-test-slave-gp6n krb5kdc[6590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:29:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:29:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:29:10.706190 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:42837
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:42837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:29:10.817615 6605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:10.817885 6605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:10.817925 6605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:10.821924 6605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:29:10.821985 6605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:10.822005 6605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:10.822021 6605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:29:10.822036 6605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:29:10.827314 6605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:42837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:42837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:10.828362 6605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:10.829396 6605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:10.837843 6611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:10.837857 6612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:10.837832 6614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:10.837971 6605 server_base.cc:1034] running on GCE node
I20260808 18:29:10.838738 6605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:10.839938 6605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:10.841120 6605 hybrid_clock.cc:648] HybridClock initialized: now 1786213750841108 us; error 30 us; skew 500 ppm
Aug 08 18:29:10 dist-test-slave-gp6n krb5kdc[6590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213750, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:10.844404 6605 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:29:10.845388 6605 webserver.cc:460] Webserver started at http://127.29.78.190:38497/ using document root <none> and password file <none>
I20260808 18:29:10.845971 6605 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:10.846074 6605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:10.846315 6605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:10.848486 6605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "ccfd6cfc5faa44b58188eb09a5f82e0a"
format_stamp: "Formatted at 2026-08-08 18:29:10 on dist-test-slave-gp6n"
server_key: "2085030a4a539dcd8bbae474e41fdb31"
server_key_iv: "36e2e471c541f109348e36cb89585455"
server_key_version: "encryptionkey@0"
I20260808 18:29:10.849092 6605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "ccfd6cfc5faa44b58188eb09a5f82e0a"
format_stamp: "Formatted at 2026-08-08 18:29:10 on dist-test-slave-gp6n"
server_key: "2085030a4a539dcd8bbae474e41fdb31"
server_key_iv: "36e2e471c541f109348e36cb89585455"
server_key_version: "encryptionkey@0"
I20260808 18:29:10.853133 6605 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260808 18:29:10.855548 6622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:10.856722 6605 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:29:10.856871 6605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "ccfd6cfc5faa44b58188eb09a5f82e0a"
format_stamp: "Formatted at 2026-08-08 18:29:10 on dist-test-slave-gp6n"
server_key: "2085030a4a539dcd8bbae474e41fdb31"
server_key_iv: "36e2e471c541f109348e36cb89585455"
server_key_version: "encryptionkey@0"
I20260808 18:29:10.856988 6605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:10.873011 6605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:10.876672 6605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:10.876910 6605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:10.884678 6605 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:42837
I20260808 18:29:10.884711 6676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:42837 every 8 connection(s)
I20260808 18:29:10.885699 6605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:29:10.888451 6677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:10.893365 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 6605
I20260808 18:29:10.893517 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:29:10.893853 30010 external_mini_cluster.cc:1428] Setting key 0aaf29206079b7e7a190ce5ece35f11b
I20260808 18:29:10.894265 6677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a: Bootstrap starting.
I20260808 18:29:10.896792 6677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:10.897553 6677 log.cc:824] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a: Log is configured to *not* fsync() on all Append() calls
I20260808 18:29:10.899441 6677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a: No bootstrap required, opened a new log
I20260808 18:29:10.903093 6677 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfd6cfc5faa44b58188eb09a5f82e0a" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42837 } }
I20260808 18:29:10.903375 6677 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:10.903439 6677 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccfd6cfc5faa44b58188eb09a5f82e0a, State: Initialized, Role: FOLLOWER
Aug 08 18:29:10 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:29:10.903934 6677 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfd6cfc5faa44b58188eb09a5f82e0a" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42837 } }
I20260808 18:29:10.904131 6677 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:29:10.904196 6677 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:29:10.904309 6677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:10.905403 6677 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfd6cfc5faa44b58188eb09a5f82e0a" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42837 } }
I20260808 18:29:10.905745 6677 leader_election.cc:302] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccfd6cfc5faa44b58188eb09a5f82e0a; no voters:
I20260808 18:29:10.906054 6677 leader_election.cc:288] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:29:10.906188 6683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:29:10.906457 6683 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [term 1 LEADER]: Becoming Leader. State: Replica: ccfd6cfc5faa44b58188eb09a5f82e0a, State: Running, Role: LEADER
I20260808 18:29:10.906810 6683 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfd6cfc5faa44b58188eb09a5f82e0a" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42837 } }
I20260808 18:29:10.907310 6677 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:29:10.908607 6684 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccfd6cfc5faa44b58188eb09a5f82e0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfd6cfc5faa44b58188eb09a5f82e0a" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42837 } } }
I20260808 18:29:10.908746 6684 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [sys.catalog]: This master's current role is: LEADER
I20260808 18:29:10.909101 6685 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccfd6cfc5faa44b58188eb09a5f82e0a. Latest consensus state: current_term: 1 leader_uuid: "ccfd6cfc5faa44b58188eb09a5f82e0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfd6cfc5faa44b58188eb09a5f82e0a" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42837 } } }
I20260808 18:29:10.909204 6685 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [sys.catalog]: This master's current role is: LEADER
I20260808 18:29:10.909487 6695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:29:10.911281 6680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:10.895344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51542 (local address 127.29.78.190:42837)
0808 18:29:10.895922 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:10.895940 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:10.895979 (+ 39us) server_negotiation.cc:407] Connection header received
0808 18:29:10.896693 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:10.896711 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:10.897118 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:10.897491 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:10.898446 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:10.899571 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:10.900535 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:10.901047 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:10.901234 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:10.904388 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:10.904395 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:10.904442 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:10.904477 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:10.906731 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:10.907321 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:10.907327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:10.907333 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:10.907494 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:10.907850 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:10.907856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:10.907858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:10.908419 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:10.909818 (+ 1399us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:10.910000 (+ 182us) server_negotiation.cc:299] Negotiation successful
0808 18:29:10.910307 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":184,"threads_started":1}
I20260808 18:29:10.913041 6695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:29:10.918725 6695 catalog_manager.cc:1347] Generated new cluster ID: 5f6705182127433fa0334806a072f97d
I20260808 18:29:10.918793 6695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:29:10.932916 6695 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:29:10.934267 6695 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:29:10.952466 6695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a: Generated new TSK 0
I20260808 18:29:10.953310 6695 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260808 18:29:10.987757 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:42837
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:29:11.109083 6708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:11.109520 6708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:11.109629 6708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:11.116901 6708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:11.116986 6708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:11.117081 6708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:29:11.125749 6708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:42837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:11.127570 6708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:11.129154 6708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:11.139441 6713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:11.139494 6717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:11.139494 6714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:11.140192 6708 server_base.cc:1034] running on GCE node
I20260808 18:29:11.140611 6708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:11.141621 6708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:11.142796 6708 hybrid_clock.cc:648] HybridClock initialized: now 1786213751142778 us; error 36 us; skew 500 ppm
Aug 08 18:29:11 dist-test-slave-gp6n krb5kdc[6590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213751, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:11.145942 6708 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:29:11.147019 6708 webserver.cc:460] Webserver started at http://127.29.78.129:33045/ using document root <none> and password file <none>
I20260808 18:29:11.147627 6708 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:11.147693 6708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:11.147961 6708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:11.150274 6708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "2d6ed672e77349878efbe0b948f3e11e"
format_stamp: "Formatted at 2026-08-08 18:29:11 on dist-test-slave-gp6n"
server_key: "7de6d9b04f1f96a268eca989b409fd57"
server_key_iv: "714dd924154dfaf3a29cbe21bfb9ccce"
server_key_version: "encryptionkey@0"
I20260808 18:29:11.150967 6708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "2d6ed672e77349878efbe0b948f3e11e"
format_stamp: "Formatted at 2026-08-08 18:29:11 on dist-test-slave-gp6n"
server_key: "7de6d9b04f1f96a268eca989b409fd57"
server_key_iv: "714dd924154dfaf3a29cbe21bfb9ccce"
server_key_version: "encryptionkey@0"
I20260808 18:29:11.155051 6708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:29:11.157574 6724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:11.158936 6708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:29:11.159080 6708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "2d6ed672e77349878efbe0b948f3e11e"
format_stamp: "Formatted at 2026-08-08 18:29:11 on dist-test-slave-gp6n"
server_key: "7de6d9b04f1f96a268eca989b409fd57"
server_key_iv: "714dd924154dfaf3a29cbe21bfb9ccce"
server_key_version: "encryptionkey@0"
I20260808 18:29:11.159199 6708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:11.176792 6708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:11.180040 6708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:11.180260 6708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:11.180892 6708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:29:11.181769 6708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:11.181844 6708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:11.181954 6708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:11.182011 6708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:11.191654 6708 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:45513
I20260808 18:29:11.191681 6848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:45513 every 8 connection(s)
I20260808 18:29:11.192580 6708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:29:11.195717 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 6708
I20260808 18:29:11.195880 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:29:11.196314 30010 external_mini_cluster.cc:1428] Setting key 57ccf39a6535bc8842c683a39e23d77d
Aug 08 18:29:11 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213751, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260808 18:29:11.207973 6680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:11.194394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:42029 (local address 127.29.78.190:42837)
0808 18:29:11.194585 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:11.194589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:11.195410 (+ 821us) server_negotiation.cc:407] Connection header received
0808 18:29:11.197306 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:11.197310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:11.197376 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:11.197464 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:11.198975 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:11.199573 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:11.200805 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:11.201084 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:11.201211 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:11.204130 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:11.204133 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:11.204135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:11.204157 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:11.205769 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:11.206290 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:11.206299 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:11.206300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:11.206382 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:11.206776 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:11.206779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:11.206781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:11.206961 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:11.207090 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:11.207665 (+ 575us) server_negotiation.cc:299] Negotiation successful
0808 18:29:11.207797 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 18:29:11.208844 6851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:11.194787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.129:42029)
0808 18:29:11.195249 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:11.195293 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:11.197073 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:11.197669 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:11.197680 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:11.198235 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:11.198809 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:11.198824 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:11.199713 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:11.199717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:11.200672 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:11.200681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:11.201250 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:11.201254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:11.201499 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:11.202193 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:11.202220 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:11.203956 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:11.205899 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:11.205904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:11.205918 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:11.206174 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:11.206518 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:11.206521 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:11.206523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:11.206669 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:11.207090 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:11.207097 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:11.207326 (+ 229us) client_negotiation.cc:769] Sending connection context
0808 18:29:11.207610 (+ 284us) client_negotiation.cc:240] Negotiation successful
0808 18:29:11.207917 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":114,"threads_started":1}
I20260808 18:29:11.210062 6849 heartbeater.cc:347] Connected to a master server at 127.29.78.190:42837
I20260808 18:29:11.210269 6849 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:11.210762 6849 heartbeater.cc:511] Master 127.29.78.190:42837 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
I20260808 18:29:11.212297 6639 ts_manager.cc:194] Registered new tserver with Master: 2d6ed672e77349878efbe0b948f3e11e (127.29.78.129:45513)
I20260808 18:29:11.213395 6639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:42029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260808 18:29:11.230319 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:42837
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:29:11.346659 6854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:11.346922 6854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:11.346999 6854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:11.351089 6854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:11.351150 6854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:11.351259 6854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:29:11.356626 6854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:42837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:11.357695 6854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:11.358842 6854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:11.368637 6859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:11.368659 6862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:11.368641 6860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:11.369288 6854 server_base.cc:1034] running on GCE node
I20260808 18:29:11.369740 6854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:11.370481 6854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:11.371668 6854 hybrid_clock.cc:648] HybridClock initialized: now 1786213751371642 us; error 35 us; skew 500 ppm
Aug 08 18:29:11 dist-test-slave-gp6n krb5kdc[6590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213751, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:11.374656 6854 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:29:11.375532 6854 webserver.cc:460] Webserver started at http://127.29.78.130:39669/ using document root <none> and password file <none>
I20260808 18:29:11.376101 6854 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:11.376166 6854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:11.376402 6854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:11.378592 6854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "1c0ee21bb45a436282e5cf1e9d0c1ec3"
format_stamp: "Formatted at 2026-08-08 18:29:11 on dist-test-slave-gp6n"
server_key: "6549579b7a95f3c56dd44b06a1ef22eb"
server_key_iv: "d08e846dc13b1b547b71fec47ccc2251"
server_key_version: "encryptionkey@0"
I20260808 18:29:11.379187 6854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "1c0ee21bb45a436282e5cf1e9d0c1ec3"
format_stamp: "Formatted at 2026-08-08 18:29:11 on dist-test-slave-gp6n"
server_key: "6549579b7a95f3c56dd44b06a1ef22eb"
server_key_iv: "d08e846dc13b1b547b71fec47ccc2251"
server_key_version: "encryptionkey@0"
I20260808 18:29:11.383100 6854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:29:11.385456 6869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:11.386847 6854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:29:11.386989 6854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "1c0ee21bb45a436282e5cf1e9d0c1ec3"
format_stamp: "Formatted at 2026-08-08 18:29:11 on dist-test-slave-gp6n"
server_key: "6549579b7a95f3c56dd44b06a1ef22eb"
server_key_iv: "d08e846dc13b1b547b71fec47ccc2251"
server_key_version: "encryptionkey@0"
I20260808 18:29:11.387112 6854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:11.424561 6854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:11.427692 6854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:11.427888 6854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:11.428471 6854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:29:11.429266 6854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:11.429347 6854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:11.429435 6854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:11.429486 6854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:11.438889 6854 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:38053
I20260808 18:29:11.438925 6997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:38053 every 8 connection(s)
I20260808 18:29:11.439759 6854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:29:11.447794 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 6854
I20260808 18:29:11.447933 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:29:11.448222 30010 external_mini_cluster.cc:1428] Setting key 4f637db150bfd9ef47fe612c8bc508c1
Aug 08 18:29:11 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213751, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:29:11.453856 6680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:11.441433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:55637 (local address 127.29.78.190:42837)
0808 18:29:11.441608 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:11.441611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:11.442424 (+ 813us) server_negotiation.cc:407] Connection header received
0808 18:29:11.443792 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:11.443795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:11.443855 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:11.443958 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:11.445101 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:11.445643 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:11.446852 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:11.447085 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:11.447196 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:11.449795 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:11.449798 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:11.449800 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:11.449822 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:11.451277 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:11.451969 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:11.451975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:11.451978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:11.452081 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:11.452469 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:11.452473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:11.452476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:11.452743 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:11.452876 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:11.453451 (+ 575us) server_negotiation.cc:299] Negotiation successful
0808 18:29:11.453615 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 18:29:11.454591 7000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:11.441816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.130:55637)
0808 18:29:11.442247 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:11.442290 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:11.443577 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:11.444140 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:11.444147 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:11.444490 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:11.444944 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:11.444956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:11.445772 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:11.445776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:11.446719 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:11.446729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:11.447235 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:11.447239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:11.447407 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:11.447997 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:11.448050 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:11.449628 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:11.451393 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:11.451402 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:11.451419 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:11.451835 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:11.452189 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:11.452192 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:11.452194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:11.452377 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:11.452946 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:11.452954 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:11.453199 (+ 245us) client_negotiation.cc:769] Sending connection context
0808 18:29:11.453373 (+ 174us) client_negotiation.cc:240] Negotiation successful
0808 18:29:11.453623 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":108,"threads_started":1}
I20260808 18:29:11.455837 6998 heartbeater.cc:347] Connected to a master server at 127.29.78.190:42837
I20260808 18:29:11.456104 6998 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:11.456730 6998 heartbeater.cc:511] Master 127.29.78.190:42837 requested a full tablet report, sending...
I20260808 18:29:11.457954 6639 ts_manager.cc:194] Registered new tserver with Master: 1c0ee21bb45a436282e5cf1e9d0c1ec3 (127.29.78.130:38053)
I20260808 18:29:11.458806 6639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:55637
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260808 18:29:11.479182 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:42837
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:29:11.589215 7003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:11.589466 7003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:11.589507 7003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:11.593442 7003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:11.593493 7003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:11.593549 7003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:29:11.598565 7003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:42837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:11.599552 7003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:11.600492 7003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:11.609935 7011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:11.609992 7008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:11.609944 7009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:11.610332 7003 server_base.cc:1034] running on GCE node
I20260808 18:29:11.610742 7003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:11.611330 7003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:11.612514 7003 hybrid_clock.cc:648] HybridClock initialized: now 1786213751612499 us; error 24 us; skew 500 ppm
Aug 08 18:29:11 dist-test-slave-gp6n krb5kdc[6590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213751, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:11.615370 7003 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:29:11.616173 7003 webserver.cc:460] Webserver started at http://127.29.78.131:46223/ using document root <none> and password file <none>
I20260808 18:29:11.616683 7003 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:11.616743 7003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:11.616895 7003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:11.618837 7003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "f028e75432cb498085d17caa89b2e147"
format_stamp: "Formatted at 2026-08-08 18:29:11 on dist-test-slave-gp6n"
server_key: "5abba25f64ee0ba0980c03b3402e1ac3"
server_key_iv: "b13ae6fb2a03a39be019c8b27d8dbf0f"
server_key_version: "encryptionkey@0"
I20260808 18:29:11.619331 7003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "f028e75432cb498085d17caa89b2e147"
format_stamp: "Formatted at 2026-08-08 18:29:11 on dist-test-slave-gp6n"
server_key: "5abba25f64ee0ba0980c03b3402e1ac3"
server_key_iv: "b13ae6fb2a03a39be019c8b27d8dbf0f"
server_key_version: "encryptionkey@0"
I20260808 18:29:11.622783 7003 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:29:11.625062 7018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:11.626210 7003 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:29:11.626325 7003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "f028e75432cb498085d17caa89b2e147"
format_stamp: "Formatted at 2026-08-08 18:29:11 on dist-test-slave-gp6n"
server_key: "5abba25f64ee0ba0980c03b3402e1ac3"
server_key_iv: "b13ae6fb2a03a39be019c8b27d8dbf0f"
server_key_version: "encryptionkey@0"
I20260808 18:29:11.626485 7003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:11.647702 7003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:11.650748 7003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:11.650936 7003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:11.651510 7003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:29:11.652328 7003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:11.652374 7003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:11.652413 7003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:11.652431 7003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:11.661988 7003 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:41811
I20260808 18:29:11.662019 7138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:41811 every 8 connection(s)
I20260808 18:29:11.662950 7003 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:29:11.665992 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 7003
I20260808 18:29:11.666143 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:29:11.666545 30010 external_mini_cluster.cc:1428] Setting key 709188754ec4218ab22629996a0430e9
Aug 08 18:29:11 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213751, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:29:11.676998 6680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:11.664837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:40367 (local address 127.29.78.190:42837)
0808 18:29:11.664977 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:11.664981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:11.665903 (+ 922us) server_negotiation.cc:407] Connection header received
0808 18:29:11.667705 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:11.667708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:11.667780 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:11.667879 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:11.669173 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:11.669793 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:11.671001 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:11.671248 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:11.671341 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:11.673754 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:11.673757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:11.673760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:11.673785 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:11.675178 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:11.675668 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:11.675670 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:11.675671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:11.675751 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:11.676067 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:11.676070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:11.676072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:11.676252 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:11.676347 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:11.676771 (+ 424us) server_negotiation.cc:299] Negotiation successful
0808 18:29:11.676872 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 18:29:11.677834 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:11.665229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.131:40367)
0808 18:29:11.665737 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:11.665798 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:11.667469 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:11.668073 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:11.668082 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:11.668430 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:11.669034 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:11.669049 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:11.669909 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:11.669911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:11.670861 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:11.670870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:11.671389 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:11.671391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:11.671524 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:11.672073 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:11.672088 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:11.673593 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:11.675295 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:11.675299 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:11.675310 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:11.675573 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:11.675831 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:11.675834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:11.675835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:11.675959 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:11.676343 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:11.676348 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:11.676569 (+ 221us) client_negotiation.cc:769] Sending connection context
0808 18:29:11.676720 (+ 151us) client_negotiation.cc:240] Negotiation successful
0808 18:29:11.676951 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":128,"threads_started":1}
I20260808 18:29:11.679054 7139 heartbeater.cc:347] Connected to a master server at 127.29.78.190:42837
I20260808 18:29:11.679304 7139 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:11.679898 7139 heartbeater.cc:511] Master 127.29.78.190:42837 requested a full tablet report, sending...
I20260808 18:29:11.680927 6639 ts_manager.cc:194] Registered new tserver with Master: f028e75432cb498085d17caa89b2e147 (127.29.78.131:41811)
I20260808 18:29:11.681499 6639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:40367
I20260808 18:29:11.692335 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.129.keytab.
Entry for principal oryx/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.129.keytab.
WARNING: no policy specified for oryx/127.29.78.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.130.keytab.
Entry for principal oryx/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.130.keytab.
WARNING: no policy specified for oryx/127.29.78.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.131.keytab.
Entry for principal oryx/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.131.keytab.
I20260808 18:29:11.770782 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 6708
I20260808 18:29:11.778383 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 6854
I20260808 18:29:11.785373 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 7003
I20260808 18:29:11.794978 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 6605
I20260808 18:29:11.802479 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:42837
--webserver_interface=127.29.78.190
--webserver_port=38497
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:42837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:29:11.917228 7148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:11.917474 7148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:11.917512 7148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:11.921411 7148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:29:11.921465 7148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:11.921485 7148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:11.921501 7148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:29:11.921516 7148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:29:11.926605 7148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:42837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:42837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=38497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:11.927529 7148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:11.928467 7148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:11.936661 7153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:11.936661 7154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:11.936817 7148 server_base.cc:1034] running on GCE node
W20260808 18:29:11.936694 7156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:11.937548 7148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:11.938639 7148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:11.939819 7148 hybrid_clock.cc:648] HybridClock initialized: now 1786213751939789 us; error 43 us; skew 500 ppm
Aug 08 18:29:11 dist-test-slave-gp6n krb5kdc[6590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213751, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:11.947897 7148 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:29:11.948797 7148 webserver.cc:460] Webserver started at http://127.29.78.190:38497/ using document root <none> and password file <none>
I20260808 18:29:11.949360 7148 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:11.949476 7148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:11.953269 7148 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:29:11.955395 7164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:11.956586 7148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:29:11.956727 7148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "ccfd6cfc5faa44b58188eb09a5f82e0a"
format_stamp: "Formatted at 2026-08-08 18:29:10 on dist-test-slave-gp6n"
server_key: "2085030a4a539dcd8bbae474e41fdb31"
server_key_iv: "36e2e471c541f109348e36cb89585455"
server_key_version: "encryptionkey@0"
I20260808 18:29:11.957262 7148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:11.970057 7148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:11.972998 7148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:11.973218 7148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:11.980760 7148 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:42837
I20260808 18:29:11.980783 7221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:42837 every 8 connection(s)
I20260808 18:29:11.981787 7148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:29:11.985970 7222 sys_catalog.cc:263] Verifying existing consensus state
I20260808 18:29:11.987661 7222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a: Bootstrap starting.
I20260808 18:29:11.988929 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 7148
I20260808 18:29:11.989390 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:45513
--local_ip_for_outbound_sockets=127.29.78.129
--tserver_master_addrs=127.29.78.190:42837
--webserver_port=33045
--webserver_interface=127.29.78.129
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.78.129
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.129.keytab with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:29:12.005805 7222 log.cc:824] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a: Log is configured to *not* fsync() on all Append() calls
I20260808 18:29:12.012135 7222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:29:12.012511 7222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a: Bootstrap complete.
I20260808 18:29:12.016768 7222 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfd6cfc5faa44b58188eb09a5f82e0a" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42837 } }
I20260808 18:29:12.017156 7222 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccfd6cfc5faa44b58188eb09a5f82e0a, State: Initialized, Role: FOLLOWER
I20260808 18:29:12.017688 7222 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfd6cfc5faa44b58188eb09a5f82e0a" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42837 } }
I20260808 18:29:12.017807 7222 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:29:12.017891 7222 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:29:12.018029 7222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [term 1 FOLLOWER]: Advancing to term 2
I20260808 18:29:12.020058 7222 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfd6cfc5faa44b58188eb09a5f82e0a" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42837 } }
I20260808 18:29:12.020385 7222 leader_election.cc:302] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccfd6cfc5faa44b58188eb09a5f82e0a; no voters:
I20260808 18:29:12.020689 7222 leader_election.cc:288] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 18:29:12.020838 7227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 18:29:12.021068 7227 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [term 2 LEADER]: Becoming Leader. State: Replica: ccfd6cfc5faa44b58188eb09a5f82e0a, State: Running, Role: LEADER
I20260808 18:29:12.021369 7227 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfd6cfc5faa44b58188eb09a5f82e0a" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42837 } }
I20260808 18:29:12.021934 7222 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:29:12.022861 7229 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccfd6cfc5faa44b58188eb09a5f82e0a. Latest consensus state: current_term: 2 leader_uuid: "ccfd6cfc5faa44b58188eb09a5f82e0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfd6cfc5faa44b58188eb09a5f82e0a" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42837 } } }
I20260808 18:29:12.022902 7228 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ccfd6cfc5faa44b58188eb09a5f82e0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfd6cfc5faa44b58188eb09a5f82e0a" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42837 } } }
I20260808 18:29:12.023013 7229 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [sys.catalog]: This master's current role is: LEADER
I20260808 18:29:12.023015 7228 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a [sys.catalog]: This master's current role is: LEADER
I20260808 18:29:12.023387 7238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:29:12.026688 7238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:29:12.027354 7238 catalog_manager.cc:1259] Loaded cluster ID: 5f6705182127433fa0334806a072f97d
I20260808 18:29:12.027415 7238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:29:12.029752 7238 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:29:12.030435 7238 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ccfd6cfc5faa44b58188eb09a5f82e0a: Loaded TSK: 0
I20260808 18:29:12.031597 7238 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 18:29:12.109395 7225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:12.109635 7225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:12.109679 7225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:12.113674 7225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:12.113731 7225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:12.113788 7225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:29:12.118973 7225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.129.keytab
--principal=oryx/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:45513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=33045
--tserver_master_addrs=127.29.78.190:42837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:12.119959 7225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:12.120970 7225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:12.131924 7252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:12.131933 7253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:12.131933 7255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:12.132123 7225 server_base.cc:1034] running on GCE node
I20260808 18:29:12.132748 7225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:12.133460 7225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:12.134642 7225 hybrid_clock.cc:648] HybridClock initialized: now 1786213752134617 us; error 42 us; skew 500 ppm
Aug 08 18:29:12 dist-test-slave-gp6n krb5kdc[6590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213752, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:12.137768 7225 init.cc:377] Logged in from keytab as oryx/127.29.78.129@KRBTEST.COM (short username oryx)
I20260808 18:29:12.138728 7225 webserver.cc:460] Webserver started at http://127.29.78.129:33045/ using document root <none> and password file <none>
I20260808 18:29:12.139314 7225 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:12.139447 7225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:12.143543 7225 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:29:12.145870 7262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:12.147275 7225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:29:12.147428 7225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "2d6ed672e77349878efbe0b948f3e11e"
format_stamp: "Formatted at 2026-08-08 18:29:11 on dist-test-slave-gp6n"
server_key: "7de6d9b04f1f96a268eca989b409fd57"
server_key_iv: "714dd924154dfaf3a29cbe21bfb9ccce"
server_key_version: "encryptionkey@0"
I20260808 18:29:12.148001 7225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:12.162771 7225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:12.166025 7225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:12.166242 7225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:12.166915 7225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:29:12.167752 7225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:12.167829 7225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:12.167922 7225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:12.167979 7225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:12.176786 7225 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:45513
I20260808 18:29:12.176843 7385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:45513 every 8 connection(s)
I20260808 18:29:12.177860 7225 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:29:12.187151 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 7225
I20260808 18:29:12.187631 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:38053
--local_ip_for_outbound_sockets=127.29.78.130
--tserver_master_addrs=127.29.78.190:42837
--webserver_port=39669
--webserver_interface=127.29.78.130
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.78.130
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.130.keytab with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 18:29:12 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:12 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
W20260808 18:29:12.192479 7388 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:12.180217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:49891 (local address 127.29.78.190:42837)
0808 18:29:12.180781 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:12.180804 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:12.181017 (+ 213us) server_negotiation.cc:407] Connection header received
0808 18:29:12.182649 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:12.182661 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:12.182957 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:12.183294 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:12.184505 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.186004 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.187219 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.187721 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.187914 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:12.191745 (+ 3831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:49891: BlockingRecv error: recv got EOF from 127.29.78.129:49891 (error 108)
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":150,"threads_started":1}
I20260808 18:29:12.192485 7389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:12.180302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.129:49891)
0808 18:29:12.180834 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:12.180890 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:12.182208 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:12.183383 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:12.183391 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:12.183820 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:12.184302 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:12.184323 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.186139 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.186143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:12.187084 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:12.187094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.188105 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.188107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:12.188242 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:12.188879 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:12.188894 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:12.191534 (+ 2640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":154,"threads_started":1}
W20260808 18:29:12.193032 7386 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:42837 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.78.190:42837: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Aug 08 18:29:12 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:12 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:12.198745 7389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:12.193522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.129:40395)
0808 18:29:12.193641 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:12.193657 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:12.193736 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:12.194006 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:12.194009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:12.194166 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:12.194251 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:12.194256 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.195083 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.195086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:12.195630 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:12.195638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.196098 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.196101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:12.196200 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:12.196688 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:12.196699 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:12.198608 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260808 18:29:12.198843 7388 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:12.193590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:40395 (local address 127.29.78.190:42837)
0808 18:29:12.193703 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:12.193708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:12.193720 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:29:12.193845 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:12.193848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:12.193893 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:12.193991 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:12.194395 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.194964 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.195751 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.195991 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.196097 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:12.198726 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:40395: BlockingRecv error: recv got EOF from 127.29.78.129:40395 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 08 18:29:12 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:12 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:12.204497 7389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:12.199278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.129:36831)
0808 18:29:12.199367 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:12.199383 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:12.199454 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:12.199679 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:12.199682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:12.199833 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:12.199904 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:12.199908 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.201024 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.201027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:12.201530 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:12.201538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.202083 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.202086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:12.202141 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:12.202617 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:12.202637 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:12.204415 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260808 18:29:12.204674 7388 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:12.199375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:36831 (local address 127.29.78.190:42837)
0808 18:29:12.199489 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:12.199495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:12.199510 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:29:12.199543 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:12.199546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:12.199587 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:12.199666 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:12.200067 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.200891 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.201633 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.201978 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.202068 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:12.204568 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:36831: BlockingRecv error: recv got EOF from 127.29.78.129:36831 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260808 18:29:12.204854 7386 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 18:29:12.308038 7391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:12.308305 7391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:12.308359 7391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:12.312773 7391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:12.312832 7391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:12.312893 7391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:29:12.318576 7391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.130.keytab
--principal=oryx/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:38053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=39669
--tserver_master_addrs=127.29.78.190:42837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:12.319568 7391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:12.320568 7391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:12.330775 7398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:12.330753 7396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:12.330843 7400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:12.331133 7391 server_base.cc:1034] running on GCE node
I20260808 18:29:12.331502 7391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:12.332144 7391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:12.333330 7391 hybrid_clock.cc:648] HybridClock initialized: now 1786213752333315 us; error 30 us; skew 500 ppm
Aug 08 18:29:12 dist-test-slave-gp6n krb5kdc[6590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213752, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:12.336185 7391 init.cc:377] Logged in from keytab as oryx/127.29.78.130@KRBTEST.COM (short username oryx)
I20260808 18:29:12.337002 7391 webserver.cc:460] Webserver started at http://127.29.78.130:39669/ using document root <none> and password file <none>
I20260808 18:29:12.337546 7391 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:12.337652 7391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:12.341454 7391 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260808 18:29:12.343556 7407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:12.344710 7391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:29:12.344856 7391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "1c0ee21bb45a436282e5cf1e9d0c1ec3"
format_stamp: "Formatted at 2026-08-08 18:29:11 on dist-test-slave-gp6n"
server_key: "6549579b7a95f3c56dd44b06a1ef22eb"
server_key_iv: "d08e846dc13b1b547b71fec47ccc2251"
server_key_version: "encryptionkey@0"
I20260808 18:29:12.345396 7391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:12.364044 7391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:12.367448 7391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:12.367655 7391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:12.368259 7391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:29:12.369082 7391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:12.369154 7391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:12.369226 7391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:12.369279 7391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:12.378265 7391 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:38053
I20260808 18:29:12.378294 7531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:38053 every 8 connection(s)
I20260808 18:29:12.379339 7391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:29:12.385430 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 7391
I20260808 18:29:12.385916 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:41811
--local_ip_for_outbound_sockets=127.29.78.131
--tserver_master_addrs=127.29.78.190:42837
--webserver_port=46223
--webserver_interface=127.29.78.131
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.78.131
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.131.keytab with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 18:29:12 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:12 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
W20260808 18:29:12.391607 7388 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:12.381234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:48441 (local address 127.29.78.190:42837)
0808 18:29:12.381379 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:12.381383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:12.382227 (+ 844us) server_negotiation.cc:407] Connection header received
0808 18:29:12.383617 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:12.383620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:12.383686 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:12.383771 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:12.384941 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.385481 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.386856 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.387088 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.387199 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:12.391423 (+ 4224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:48441: BlockingRecv error: recv got EOF from 127.29.78.130:48441 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 18:29:12.392374 7534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:12.381621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.130:48441)
0808 18:29:12.382066 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:12.382110 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:12.383384 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:12.383946 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:12.383953 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:12.384315 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:12.384790 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:12.384804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.385621 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.385626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:12.386727 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:12.386737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.387257 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.387261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:12.387429 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:12.388099 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:12.388125 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:12.391233 (+ 3108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
W20260808 18:29:12.393455 7532 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:42837 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.78.190:42837: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Aug 08 18:29:12 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:12 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:12.399160 7534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:12.393939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.130:57685)
0808 18:29:12.394055 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:12.394071 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:12.394153 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:12.394418 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:12.394422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:12.394599 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:12.394688 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:12.394694 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.395421 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.395428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:12.396005 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:12.396013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.396473 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.396477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:12.396579 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:12.397094 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:12.397108 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:12.399023 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260808 18:29:12.399259 7388 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:12.394013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:57685 (local address 127.29.78.190:42837)
0808 18:29:12.394134 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:12.394138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:12.394150 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:29:12.394240 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:12.394243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:12.394284 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:12.394386 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:12.394801 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.395313 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.396129 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.396362 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.396471 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:12.399146 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:57685: BlockingRecv error: recv got EOF from 127.29.78.130:57685 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 08 18:29:12 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:12 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:12.404851 7534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:12.399692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.130:44327)
0808 18:29:12.399819 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:12.399835 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:12.399910 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:12.400129 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:12.400132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:12.400294 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:12.400383 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:12.400388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.401080 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.401083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:12.401956 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:12.401966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.402374 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.402378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:12.402491 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:12.402921 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:12.402935 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:12.404735 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 18:29:12.404980 7388 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:12.399763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:44327 (local address 127.29.78.190:42837)
0808 18:29:12.399862 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:12.399865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:12.399876 (+ 11us) server_negotiation.cc:407] Connection header received
0808 18:29:12.399993 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:12.399995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:12.400027 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:12.400092 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:12.400500 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.400980 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.402070 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.402271 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.402350 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:12.404848 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:44327: BlockingRecv error: recv got EOF from 127.29.78.130:44327 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260808 18:29:12.405323 7532 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 18:29:12.501157 7535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:12.501498 7535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:12.501577 7535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:12.505646 7535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:12.505726 7535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:12.505817 7535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:29:12.510968 7535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/krb5kdc/oryx_127.29.78.131.keytab
--principal=oryx/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:41811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=46223
--tserver_master_addrs=127.29.78.190:42837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:12.511983 7535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:12.512992 7535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:12.522455 7543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:12.522501 7541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:12.522639 7535 server_base.cc:1034] running on GCE node
W20260808 18:29:12.522804 7540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:12.523226 7535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:12.523876 7535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:12.525066 7535 hybrid_clock.cc:648] HybridClock initialized: now 1786213752525049 us; error 29 us; skew 500 ppm
Aug 08 18:29:12 dist-test-slave-gp6n krb5kdc[6590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213752, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:12.527938 7535 init.cc:377] Logged in from keytab as oryx/127.29.78.131@KRBTEST.COM (short username oryx)
I20260808 18:29:12.528782 7535 webserver.cc:460] Webserver started at http://127.29.78.131:46223/ using document root <none> and password file <none>
I20260808 18:29:12.529341 7535 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:12.529445 7535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:12.533247 7535 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:29:12.535280 7550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:12.536456 7535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 18:29:12.536586 7535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "f028e75432cb498085d17caa89b2e147"
format_stamp: "Formatted at 2026-08-08 18:29:11 on dist-test-slave-gp6n"
server_key: "5abba25f64ee0ba0980c03b3402e1ac3"
server_key_iv: "b13ae6fb2a03a39be019c8b27d8dbf0f"
server_key_version: "encryptionkey@0"
I20260808 18:29:12.537118 7535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:12.555418 7535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:12.558267 7535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:12.558529 7535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:12.559134 7535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:29:12.559962 7535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:12.560032 7535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:12.560101 7535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:12.560143 7535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:12.569325 7535 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:41811
I20260808 18:29:12.569341 7678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:41811 every 8 connection(s)
I20260808 18:29:12.570338 7535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:29:12.574195 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 7535
Aug 08 18:29:12 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:12 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
W20260808 18:29:12.582765 7388 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:12.572319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:40493 (local address 127.29.78.190:42837)
0808 18:29:12.572452 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:12.572455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:12.573273 (+ 818us) server_negotiation.cc:407] Connection header received
0808 18:29:12.574755 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:12.574759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:12.574817 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:12.574891 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:12.576558 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.577122 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.578386 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.578748 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.578856 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:12.582663 (+ 3807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:40493: BlockingRecv error: recv got EOF from 127.29.78.131:40493 (error 108)
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":47}
I20260808 18:29:12.583654 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:12.572664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.131:40493)
0808 18:29:12.573110 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:12.573153 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:12.574524 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:12.575398 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:12.575410 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:12.575915 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:12.576384 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:12.576398 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.577281 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.577284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:12.578267 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:12.578277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.578943 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.578948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:12.579167 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:12.579705 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:12.579720 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:12.582560 (+ 2840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":118,"threads_started":1}
W20260808 18:29:12.584196 7680 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:42837 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.78.190:42837: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
I20260808 18:29:12.584748 7684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:12.575243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52786 (local address 127.29.78.190:42837)
0808 18:29:12.575526 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:12.575530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:12.575545 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:29:12.575588 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:12.575591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:12.575632 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:12.575732 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:12.576518 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.577111 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.578037 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.578379 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.578517 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:12.580004 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:12.580011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:12.580037 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:12.580067 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:12.582110 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:12.582618 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:12.582625 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:12.582631 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:12.582784 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:12.583118 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:12.583121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:12.583123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:12.583545 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:12.583745 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:12.584397 (+ 652us) server_negotiation.cc:299] Negotiation successful
0808 18:29:12.584608 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":209,"thread_start_us":105,"threads_started":1}
Aug 08 18:29:12 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:12 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:12.590329 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:12.584601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.131:41133)
0808 18:29:12.585492 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:12.585505 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:12.585606 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:12.585850 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:12.585852 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:12.586007 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:12.586114 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:12.586121 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.587084 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.587086 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:12.587721 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:12.587730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.588118 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.588121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:12.588185 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:12.588641 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:12.588652 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:12.590247 (+ 1595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161}
W20260808 18:29:12.590554 7388 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:12.584708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:41133 (local address 127.29.78.190:42837)
0808 18:29:12.584803 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:12.584807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:12.585569 (+ 762us) server_negotiation.cc:407] Connection header received
0808 18:29:12.585711 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:12.585714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:12.585756 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:12.585836 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:12.586257 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.586978 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.587822 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.588020 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.588103 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:12.590381 (+ 2278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:41133: BlockingRecv error: recv got EOF from 127.29.78.131:41133 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Aug 08 18:29:12 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:12 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:12.595520 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:12.590991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.131:46187)
0808 18:29:12.591119 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:12.591135 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:12.591211 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:12.591440 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:12.591442 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:12.591591 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:12.591655 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:12.591659 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.592330 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.592336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:12.592840 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:12.592847 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.593272 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.593275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:12.593354 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:12.593735 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:12.593745 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:12.595389 (+ 1644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 18:29:12.595606 7388 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:12.591040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:46187 (local address 127.29.78.190:42837)
0808 18:29:12.591190 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:12.591195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:12.591207 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:29:12.591301 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:12.591304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:12.591345 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:12.591425 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:12.591762 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.592228 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.592955 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:12.593165 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:12.593272 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:12.595493 (+ 2221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:46187: BlockingRecv error: recv got EOF from 127.29.78.131:46187 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260808 18:29:12.595777 7680 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 08 18:29:13 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:13 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:13.211301 7685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:13.205433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.129:40907)
0808 18:29:13.205710 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:13.205723 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:13.205818 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:13.206147 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:13.206154 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:13.206350 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:13.206535 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:13.206542 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:13.207284 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:13.207287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:13.207930 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:13.207939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:13.208362 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:13.208365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:13.208451 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:13.208985 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:13.208997 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:13.211180 (+ 2183us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":98,"threads_started":1}
W20260808 18:29:13.211483 7686 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:13.205613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:40907 (local address 127.29.78.190:42837)
0808 18:29:13.205878 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:13.205881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:13.205893 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:29:13.205928 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:13.205930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:13.205971 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:13.206064 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:13.206653 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:13.207193 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:13.208042 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:13.208261 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:13.208336 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:13.211310 (+ 2974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:40907: BlockingRecv error: recv got EOF from 127.29.78.129:40907 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":119,"threads_started":1}
Aug 08 18:29:13 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:13 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:13.411747 7687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:13.405993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.130:44407)
0808 18:29:13.406300 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:13.406315 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:13.406472 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:13.406812 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:13.406815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:13.407024 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:13.407137 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:13.407143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:13.407934 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:13.407940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:13.408519 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:13.408527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:13.408996 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:13.409000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:13.409097 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:13.409617 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:13.409628 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:13.411627 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":135,"threads_started":1}
W20260808 18:29:13.411888 7686 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:13.406066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:44407 (local address 127.29.78.190:42837)
0808 18:29:13.406233 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:13.406238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:13.406388 (+ 150us) server_negotiation.cc:407] Connection header received
0808 18:29:13.406645 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:13.406648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:13.406704 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:13.406794 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:13.407255 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:13.407820 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:13.408640 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:13.408879 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:13.408992 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:13.411770 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:44407: BlockingRecv error: recv got EOF from 127.29.78.130:44407 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 08 18:29:13 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:13 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:13.602341 7688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:13.596436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.131:41381)
0808 18:29:13.596746 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:13.596759 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:13.596884 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:13.597167 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:13.597170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:13.597421 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:13.597517 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:13.597523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:13.598310 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:13.598316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:13.598935 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:13.598943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:13.599412 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:13.599415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:13.599504 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:13.600087 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:13.600102 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:13.602187 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":103,"threads_started":1}
W20260808 18:29:13.602495 7686 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:13.596512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:41381 (local address 127.29.78.190:42837)
0808 18:29:13.596683 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:13.596689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:13.596836 (+ 147us) server_negotiation.cc:407] Connection header received
0808 18:29:13.596992 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:13.596995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:13.597052 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:13.597143 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:13.597642 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:13.598182 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:13.599059 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:13.599293 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:13.599408 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:13.602337 (+ 2929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:41381: BlockingRecv error: recv got EOF from 127.29.78.131:41381 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 08 18:29:14 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:14 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:14.217828 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:14.212132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.129:34927)
0808 18:29:14.212457 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:14.212472 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:14.212572 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:14.212860 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:14.212863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:14.213057 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:14.213173 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:14.213179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:14.213907 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:14.213910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:14.214515 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:14.214525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:14.214957 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:14.214961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:14.215028 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:14.215605 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:14.215617 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:14.217632 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":112,"threads_started":1}
W20260808 18:29:14.217996 7690 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:14.212187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:34927 (local address 127.29.78.190:42837)
0808 18:29:14.212475 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:14.212478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:14.212522 (+ 44us) server_negotiation.cc:407] Connection header received
0808 18:29:14.212658 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:14.212661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:14.212706 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:14.212787 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:14.213281 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:14.213800 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:14.214630 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:14.214854 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:14.214933 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:14.217770 (+ 2837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:34927: BlockingRecv error: recv got EOF from 127.29.78.129:34927 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":110,"threads_started":1}
Aug 08 18:29:14 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:14 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:14.418519 7691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:14.412761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.130:45889)
0808 18:29:14.413101 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:14.413116 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:14.413218 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:14.413492 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:14.413495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:14.413691 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:14.413785 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:14.413790 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:14.414716 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:14.414718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:14.415273 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:14.415280 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:14.415761 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:14.415763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:14.415823 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:14.416349 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:14.416359 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:14.418320 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":134,"threads_started":1}
W20260808 18:29:14.418749 7690 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:14.412826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:45889 (local address 127.29.78.190:42837)
0808 18:29:14.412995 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:14.413000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:14.413191 (+ 191us) server_negotiation.cc:407] Connection header received
0808 18:29:14.413298 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:14.413301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:14.413381 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:14.413459 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:14.413915 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:14.414616 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:14.415420 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:14.415660 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:14.415747 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:14.418601 (+ 2854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:45889: BlockingRecv error: recv got EOF from 127.29.78.130:45889 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 08 18:29:14 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:14 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:14.609606 7692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:14.603277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.131:42463)
0808 18:29:14.603708 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:14.603725 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:14.603853 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:14.604127 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:14.604130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:14.604369 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:14.604466 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:14.604473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:14.605301 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:14.605307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:14.606116 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:14.606126 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:14.606715 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:14.606719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:14.606817 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:14.607421 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:14.607434 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:14.609461 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":117,"threads_started":1}
W20260808 18:29:14.609694 7690 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:14.603427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:42463 (local address 127.29.78.190:42837)
0808 18:29:14.603583 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:14.603588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:14.603807 (+ 219us) server_negotiation.cc:407] Connection header received
0808 18:29:14.603952 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:14.603956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:14.604007 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:14.604110 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:14.604595 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:14.605183 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:14.606246 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:14.606594 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:14.606715 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:14.609602 (+ 2887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:42463: BlockingRecv error: recv got EOF from 127.29.78.131:42463 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 08 18:29:15 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:15 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:15.225351 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:15.219102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.129:42291)
0808 18:29:15.219415 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:15.219430 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:15.219524 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:15.219794 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:15.219796 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:15.219980 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:15.220087 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:15.220092 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:15.220903 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:15.220905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:15.221468 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:15.221475 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:15.222176 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:15.222179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:15.222247 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:15.222836 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:15.222847 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:15.225233 (+ 2386us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":100,"threads_started":1}
W20260808 18:29:15.225569 7694 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:15.219185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:42291 (local address 127.29.78.190:42837)
0808 18:29:15.219485 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:15.219489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:15.219504 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:29:15.219628 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:15.219630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:15.219676 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:15.219760 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:15.220206 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:15.220805 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:15.221777 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:15.222060 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:15.222152 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:15.225375 (+ 3223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:42291: BlockingRecv error: recv got EOF from 127.29.78.129:42291 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":95,"threads_started":1}
Aug 08 18:29:15 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:15 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:15.425091 7695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:15.419241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.130:47189)
0808 18:29:15.419579 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:15.419593 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:15.419722 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:15.419990 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:15.419993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:15.420211 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:15.420302 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:15.420308 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:15.421097 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:15.421100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:15.421739 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:15.421747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:15.422213 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:15.422216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:15.422304 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:15.422943 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:15.422955 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:15.424907 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":102,"threads_started":1}
W20260808 18:29:15.425134 7694 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:15.419331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:47189 (local address 127.29.78.190:42837)
0808 18:29:15.419490 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:15.419495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:15.419666 (+ 171us) server_negotiation.cc:407] Connection header received
0808 18:29:15.419827 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:15.419830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:15.419882 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:15.419971 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:15.420417 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:15.420946 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:15.421857 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:15.422093 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:15.422201 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:15.425032 (+ 2831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:47189: BlockingRecv error: recv got EOF from 127.29.78.130:47189 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 08 18:29:15 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:15 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:15.615943 7696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:15.610617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.131:38257)
0808 18:29:15.610901 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:15.610915 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:15.611011 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:15.611304 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:15.611307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:15.611517 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:15.611617 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:15.611622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:15.612342 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:15.612344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:15.612885 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:15.612892 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:15.613301 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:15.613303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:15.613368 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:15.613846 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:15.613857 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:15.615831 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":100,"threads_started":1}
W20260808 18:29:15.616108 7694 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:15.610662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:38257 (local address 127.29.78.190:42837)
0808 18:29:15.610798 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:15.610802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:15.610987 (+ 185us) server_negotiation.cc:407] Connection header received
0808 18:29:15.611102 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:15.611104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:15.611176 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:15.611266 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:15.611722 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:15.612249 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:15.612982 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:15.613206 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:15.613282 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:15.615981 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:38257: BlockingRecv error: recv got EOF from 127.29.78.131:38257 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 08 18:29:16 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:16 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:16.231945 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:16.226182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.129:58763)
0808 18:29:16.226741 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:16.226757 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:16.226882 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:16.227231 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:16.227236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:16.227475 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:16.227614 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:16.227624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:16.228416 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:16.228419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:16.228972 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:16.228980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:16.229395 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:16.229397 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:16.229458 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:16.229968 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:16.229982 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:16.231832 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":99,"threads_started":1}
W20260808 18:29:16.232100 7699 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:16.226256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:58763 (local address 127.29.78.190:42837)
0808 18:29:16.226607 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:16.226612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:16.226843 (+ 231us) server_negotiation.cc:407] Connection header received
0808 18:29:16.227008 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:16.227013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:16.227069 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:16.227170 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:16.227755 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:16.228307 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:16.229068 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:16.229287 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:16.229410 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:16.231966 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:58763: BlockingRecv error: recv got EOF from 127.29.78.129:58763 (error 108)
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":83,"threads_started":1}
Aug 08 18:29:16 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:16 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:16.431865 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:16.426179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.130:43471)
0808 18:29:16.426528 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:16.426546 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:16.426688 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:16.427047 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:16.427049 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:16.427228 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:16.427321 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:16.427326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:16.428065 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:16.428067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:16.428606 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:16.428613 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:16.429066 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:16.429069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:16.429133 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:16.429616 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:16.429626 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:16.431687 (+ 2061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":105,"threads_started":1}
W20260808 18:29:16.432168 7699 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:16.426220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:43471 (local address 127.29.78.190:42837)
0808 18:29:16.426382 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:16.426386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:16.426636 (+ 250us) server_negotiation.cc:407] Connection header received
0808 18:29:16.426866 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:16.426869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:16.426924 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:16.427000 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:16.427419 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:16.427960 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:16.428710 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:16.428951 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:16.429037 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:16.431837 (+ 2800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:43471: BlockingRecv error: recv got EOF from 127.29.78.130:43471 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 18:29:16 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:16 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:16.622455 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:16.616818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.131:37621)
0808 18:29:16.617114 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:16.617128 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:16.617226 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:16.617457 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:16.617459 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:16.617654 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:16.617749 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:16.617755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:16.618548 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:16.618552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:16.619136 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:16.619145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:16.619628 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:16.619632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:16.619726 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:16.620269 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:16.620282 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:16.622255 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":114,"threads_started":1}
W20260808 18:29:16.622587 7699 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:16.616848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:37621 (local address 127.29.78.190:42837)
0808 18:29:16.617030 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:16.617035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:16.617202 (+ 167us) server_negotiation.cc:407] Connection header received
0808 18:29:16.617307 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:16.617309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:16.617357 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:16.617425 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:16.617847 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:16.618375 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:16.619264 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:16.619501 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:16.619610 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:16.622431 (+ 2821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:37621: BlockingRecv error: recv got EOF from 127.29.78.131:37621 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 08 18:29:17 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:17 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:17.244033 7703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:17.232919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.129:48835)
0808 18:29:17.233198 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:17.233218 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:17.233314 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:17.233553 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:17.233555 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:17.233752 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:17.233863 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:17.233869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:17.234657 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:17.234661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:17.235241 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:17.235251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:17.235685 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:17.235688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:17.235762 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:17.241878 (+ 6116us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:17.241891 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:17.243911 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":96,"threads_started":1}
W20260808 18:29:17.244197 7704 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:17.233049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:48835 (local address 127.29.78.190:42837)
0808 18:29:17.233300 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:17.233304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:17.233318 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:29:17.233392 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:17.233394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:17.233435 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:17.233514 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:17.233978 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:17.234537 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:17.235349 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:17.235562 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:17.235637 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:17.244063 (+ 8426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:48835: BlockingRecv error: recv got EOF from 127.29.78.129:48835 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":97,"threads_started":1}
Aug 08 18:29:17 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:17 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:17.438352 7705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:17.432815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.130:40893)
0808 18:29:17.433066 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:17.433088 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:17.433190 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:17.433455 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:17.433457 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:17.433660 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:17.433751 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:17.433756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:17.434549 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:17.434553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:17.435133 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:17.435142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:17.435702 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:17.435706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:17.435796 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:17.436367 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:17.436380 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:17.438240 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":85,"threads_started":1}
W20260808 18:29:17.438555 7704 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:17.432951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:40893 (local address 127.29.78.190:42837)
0808 18:29:17.433090 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:17.433093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:17.433161 (+ 68us) server_negotiation.cc:407] Connection header received
0808 18:29:17.433300 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:17.433302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:17.433347 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:17.433423 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:17.433849 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:17.434376 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:17.435258 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:17.435583 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:17.435695 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:17.438370 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:40893: BlockingRecv error: recv got EOF from 127.29.78.130:40893 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 18:29:17 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:17 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:17.629592 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:17.623335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.131:36673)
0808 18:29:17.623673 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:17.623688 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:17.623806 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:17.624125 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:17.624128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:17.624369 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:17.624461 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:17.624467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:17.625311 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:17.625317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:17.625960 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:17.625969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:17.626535 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:17.626539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:17.626634 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:17.627259 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:17.627273 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:17.629415 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":114,"threads_started":1}
W20260808 18:29:17.629655 7704 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:17.623450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:36673 (local address 127.29.78.190:42837)
0808 18:29:17.623635 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:17.623640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:17.623769 (+ 129us) server_negotiation.cc:407] Connection header received
0808 18:29:17.623943 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:17.623948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:17.624008 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:17.624103 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:17.624588 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:17.625190 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:17.626094 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:17.626360 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:17.626533 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:17.629542 (+ 3009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:36673: BlockingRecv error: recv got EOF from 127.29.78.131:36673 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 08 18:29:18 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:18 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:18.250298 7707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:18.244807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.129:42833)
0808 18:29:18.245129 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:18.245143 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:18.245248 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:18.245477 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:18.245479 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:18.245670 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:18.245767 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:18.245773 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:18.246570 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:18.246572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:18.247139 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:18.247147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:18.247556 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:18.247559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:18.247623 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:18.248094 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:18.248106 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:18.250185 (+ 2079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":93,"threads_started":1}
W20260808 18:29:18.250542 7708 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:18.244857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:42833 (local address 127.29.78.190:42837)
0808 18:29:18.245141 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:18.245145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:18.245197 (+ 52us) server_negotiation.cc:407] Connection header received
0808 18:29:18.245326 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:18.245328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:18.245369 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:18.245439 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:18.245881 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:18.246477 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:18.247235 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:18.247458 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:18.247532 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:18.250313 (+ 2781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:42833: BlockingRecv error: recv got EOF from 127.29.78.129:42833 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":98,"threads_started":1}
Aug 08 18:29:18 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:18 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:18.444975 7709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:18.439237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.130:56171)
0808 18:29:18.439537 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:18.439552 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:18.439690 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:18.439980 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:18.439983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:18.440204 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:18.440312 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:18.440318 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:18.441184 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:18.441187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:18.441854 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:18.441863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:18.442427 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:18.442431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:18.442501 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:18.443019 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:18.443043 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:18.444848 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":98,"threads_started":1}
W20260808 18:29:18.445113 7708 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:18.439315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:56171 (local address 127.29.78.190:42837)
0808 18:29:18.439499 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:18.439504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:18.439634 (+ 130us) server_negotiation.cc:407] Connection header received
0808 18:29:18.439797 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:18.439801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:18.439855 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:18.439941 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:18.440460 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:18.441058 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:18.441962 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:18.442206 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:18.442297 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:18.444994 (+ 2697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:56171: BlockingRecv error: recv got EOF from 127.29.78.130:56171 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 08 18:29:18 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:18 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:18.636124 7710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:18.630418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.131:60589)
0808 18:29:18.630753 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:18.630766 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:18.630874 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:18.631107 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:18.631109 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:18.631300 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:18.631403 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:18.631408 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:18.632119 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:18.632121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:18.632704 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:18.632712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:18.633112 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:18.633115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:18.633179 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:18.633753 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:18.633769 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:18.636004 (+ 2235us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":123,"threads_started":1}
W20260808 18:29:18.636257 7708 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:18.630516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:60589 (local address 127.29.78.190:42837)
0808 18:29:18.630674 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:18.630679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:18.630840 (+ 161us) server_negotiation.cc:407] Connection header received
0808 18:29:18.630955 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:18.630957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:18.631003 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:18.631071 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:18.631504 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:18.632027 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:18.632806 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:18.633012 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:18.633084 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:18.636138 (+ 3054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:60589: BlockingRecv error: recv got EOF from 127.29.78.131:60589 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 08 18:29:19 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:19 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:19.257385 7711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:19.251165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.129:35835)
0808 18:29:19.251585 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:19.251603 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:19.251736 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:19.252091 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:19.252095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:19.252362 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:19.252494 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:19.252539 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:19.253550 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:19.253552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:19.254115 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:19.254123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:19.254805 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:19.254807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:19.254891 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:19.255439 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:19.255460 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:19.257271 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":106,"threads_started":1}
W20260808 18:29:19.257629 7713 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:19.251249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:35835 (local address 127.29.78.190:42837)
0808 18:29:19.251665 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:19.251670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:19.251688 (+ 18us) server_negotiation.cc:407] Connection header received
0808 18:29:19.251862 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:19.251865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:19.251937 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:19.252047 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:19.252686 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:19.253430 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:19.254248 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:19.254698 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:19.254793 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:19.257413 (+ 2620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:35835: BlockingRecv error: recv got EOF from 127.29.78.129:35835 (error 108)
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":83,"threads_started":1}
Aug 08 18:29:19 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:19 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:19.452247 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:19.445819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.130:40455)
0808 18:29:19.446374 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:19.446388 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:19.446516 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:19.446770 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:19.446772 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:19.446975 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:19.447070 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:19.447076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:19.447898 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:19.447901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:19.448893 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:19.448903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:19.449536 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:19.449539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:19.449607 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:19.450176 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:19.450188 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:19.452135 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":144,"threads_started":1}
W20260808 18:29:19.452464 7713 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:19.445900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:40455 (local address 127.29.78.190:42837)
0808 18:29:19.446101 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:19.446106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:19.446513 (+ 407us) server_negotiation.cc:407] Connection header received
0808 18:29:19.446612 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:19.446614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:19.446662 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:19.446736 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:19.447203 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:19.447756 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:19.448995 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:19.449404 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:19.449501 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:19.452316 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:40455: BlockingRecv error: recv got EOF from 127.29.78.130:40455 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 08 18:29:19 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:19 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:19.642562 7715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:19.637038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.131:57669)
0808 18:29:19.637359 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:19.637373 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:19.637475 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:19.637711 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:19.637714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:19.637906 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:19.638001 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:19.638006 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:19.638788 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:19.638791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:19.639349 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:19.639357 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:19.639786 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:19.639789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:19.639847 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:19.640470 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:19.640490 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:19.642386 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":93,"threads_started":1}
W20260808 18:29:19.642728 7713 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:19.637131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:57669 (local address 127.29.78.190:42837)
0808 18:29:19.637298 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:19.637302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:19.637468 (+ 166us) server_negotiation.cc:407] Connection header received
0808 18:29:19.637559 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:19.637561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:19.637609 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:19.637672 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:19.638137 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:19.638689 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:19.639460 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:19.639688 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:19.639768 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:19.642573 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:57669: BlockingRecv error: recv got EOF from 127.29.78.131:57669 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 08 18:29:20 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:20 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:20.264050 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:20.258182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.129:33033)
0808 18:29:20.258511 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:20.258525 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:20.258616 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:20.258996 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:20.259002 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:20.259205 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:20.259331 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:20.259339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:20.260405 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:20.260408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:20.260974 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:20.260985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:20.261496 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:20.261499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:20.261567 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:20.262073 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:20.262084 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:20.263932 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":128,"threads_started":1}
W20260808 18:29:20.264197 7717 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:20.258343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:33033 (local address 127.29.78.190:42837)
0808 18:29:20.258705 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:20.258710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:20.258727 (+ 17us) server_negotiation.cc:407] Connection header received
0808 18:29:20.258771 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:20.258774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:20.258831 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:20.258939 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:20.259474 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:20.260298 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:20.261076 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:20.261362 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:20.261450 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:20.264063 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:33033: BlockingRecv error: recv got EOF from 127.29.78.129:33033 (error 108)
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":101,"threads_started":1}
Aug 08 18:29:20 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:20 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:20.459125 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:20.452981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.130:37371)
0808 18:29:20.453355 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:20.453367 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:20.453502 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:20.453773 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:20.453776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:20.453994 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:20.454087 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:20.454093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:20.455080 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:20.455086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:20.455827 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:20.455839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:20.456361 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:20.456364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:20.456453 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:20.457069 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:20.457089 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:20.459008 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":122,"threads_started":1}
W20260808 18:29:20.459259 7717 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:20.453074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:37371 (local address 127.29.78.190:42837)
0808 18:29:20.453250 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:20.453255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:20.453450 (+ 195us) server_negotiation.cc:407] Connection header received
0808 18:29:20.453605 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:20.453608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:20.453662 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:20.453749 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:20.454212 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:20.454970 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:20.455966 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:20.456248 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:20.456347 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:20.459141 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:37371: BlockingRecv error: recv got EOF from 127.29.78.130:37371 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 08 18:29:20 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:20 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:20.648845 7719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:20.643279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.131:60331)
0808 18:29:20.643580 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:20.643592 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:20.643729 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:20.643995 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:20.643998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:20.644218 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:20.644310 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:20.644316 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:20.645083 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:20.645088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:20.645660 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:20.645667 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:20.646143 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:20.646147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:20.646235 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:20.646764 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:20.646775 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:20.648720 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":87,"threads_started":1}
W20260808 18:29:20.648958 7717 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:20.643362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:60331 (local address 127.29.78.190:42837)
0808 18:29:20.643522 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:20.643527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:20.643666 (+ 139us) server_negotiation.cc:407] Connection header received
0808 18:29:20.643827 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:20.643831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:20.643880 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:20.643974 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:20.644431 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:20.644963 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:20.645782 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:20.646021 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:20.646140 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:20.648843 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:60331: BlockingRecv error: recv got EOF from 127.29.78.131:60331 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 08 18:29:21 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:21 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:21.270836 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:21.264815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.129:33441)
0808 18:29:21.265253 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:21.265272 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:21.265388 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:21.265768 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:21.265773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:21.266025 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:21.266156 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:21.266179 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:21.267024 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:21.267027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:21.267606 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:21.267614 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:21.268035 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:21.268037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:21.268098 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:21.268562 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:21.268573 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:21.270633 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":116,"threads_started":1}
W20260808 18:29:21.270874 7721 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:21.264949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:33441 (local address 127.29.78.190:42837)
0808 18:29:21.265192 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:21.265196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:21.265357 (+ 161us) server_negotiation.cc:407] Connection header received
0808 18:29:21.265548 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:21.265553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:21.265621 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:21.265707 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:21.266302 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:21.266927 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:21.267716 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:21.267939 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:21.268016 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:21.270752 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:33441: BlockingRecv error: recv got EOF from 127.29.78.129:33441 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":79,"threads_started":1}
Aug 08 18:29:21 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:21 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:21.465898 7722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:21.459929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.130:48443)
0808 18:29:21.460221 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:21.460234 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:21.460364 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:21.460603 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:21.460606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:21.460790 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:21.460884 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:21.460889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:21.461866 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:21.461871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:21.462766 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:21.462777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:21.463189 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:21.463192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:21.463255 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:21.463731 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:21.463743 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:21.465703 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":132,"threads_started":1}
W20260808 18:29:21.465919 7721 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:21.460029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:48443 (local address 127.29.78.190:42837)
0808 18:29:21.460191 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:21.460195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:21.460307 (+ 112us) server_negotiation.cc:407] Connection header received
0808 18:29:21.460443 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:21.460445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:21.460497 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:21.460568 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:21.460984 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:21.461744 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:21.462875 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:21.463095 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:21.463176 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:21.465818 (+ 2642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:48443: BlockingRecv error: recv got EOF from 127.29.78.130:48443 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 08 18:29:21 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:21 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:21.655467 7723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:21.649764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.131:54961)
0808 18:29:21.650065 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:21.650080 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:21.650220 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:21.650513 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:21.650515 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:21.650740 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:21.650832 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:21.650838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:21.651612 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:21.651616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:21.652212 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:21.652220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:21.652683 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:21.652687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:21.652775 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:21.653307 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:21.653319 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:21.655330 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":91,"threads_started":1}
W20260808 18:29:21.655637 7721 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:21.649828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:54961 (local address 127.29.78.190:42837)
0808 18:29:21.650010 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:21.650015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:21.650165 (+ 150us) server_negotiation.cc:407] Connection header received
0808 18:29:21.650328 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:21.650333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:21.650385 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:21.650492 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:21.650949 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:21.651497 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:21.652337 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:21.652570 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:21.652680 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:21.655472 (+ 2792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:54961: BlockingRecv error: recv got EOF from 127.29.78.131:54961 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 08 18:29:22 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:22 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:22.280711 7724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:22.271680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.129:44111)
0808 18:29:22.271956 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:22.271969 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:22.272057 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:22.272308 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:22.272310 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:22.272500 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:22.272610 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:22.272616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:22.273373 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:22.273375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:22.273928 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:22.273935 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:22.274337 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:22.274340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:22.274452 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:22.278646 (+ 4194us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:22.278658 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:22.280558 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":100,"threads_started":1}
W20260808 18:29:22.281018 7725 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:22.271725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:44111 (local address 127.29.78.190:42837)
0808 18:29:22.272008 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:22.272011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:22.272024 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:29:22.272147 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:22.272149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:22.272192 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:22.272270 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:22.272726 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:22.273280 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:22.274027 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:22.274242 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:22.274315 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:22.280726 (+ 6411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:44111: BlockingRecv error: recv got EOF from 127.29.78.129:44111 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":123,"threads_started":1}
Aug 08 18:29:22 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:22 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:22.471807 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:22.466762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.130:36089)
0808 18:29:22.467026 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:22.467042 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:22.467138 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:22.467371 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:22.467373 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:22.467573 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:22.467667 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:22.467672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:22.468390 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:22.468392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:22.468946 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:22.468953 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:22.469360 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:22.469363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:22.469425 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:22.469918 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:22.469929 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:22.471689 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":73,"threads_started":1}
W20260808 18:29:22.471994 7725 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:22.466857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:36089 (local address 127.29.78.190:42837)
0808 18:29:22.467052 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:22.467056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:22.467118 (+ 62us) server_negotiation.cc:407] Connection header received
0808 18:29:22.467222 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:22.467224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:22.467268 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:22.467334 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:22.467766 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:22.468303 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:22.469046 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:22.469260 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:22.469346 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:22.471828 (+ 2482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:36089: BlockingRecv error: recv got EOF from 127.29.78.130:36089 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 08 18:29:22 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:22 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:22.661463 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:22.656226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.131:32829)
0808 18:29:22.656524 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:22.656538 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:22.656634 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:22.656865 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:22.656867 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:22.657063 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:22.657158 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:22.657164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:22.657879 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:22.657881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:22.658514 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:22.658524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:22.658927 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:22.658930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:22.658989 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:22.659483 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:22.659494 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:22.661298 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":86,"threads_started":1}
W20260808 18:29:22.661845 7725 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:22.656402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:32829 (local address 127.29.78.190:42837)
0808 18:29:22.656527 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:22.656531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:22.656607 (+ 76us) server_negotiation.cc:407] Connection header received
0808 18:29:22.656713 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:22.656716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:22.656762 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:22.656829 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:22.657259 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:22.657789 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:22.658616 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:22.658827 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:22.658911 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:22.661458 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:32829: BlockingRecv error: recv got EOF from 127.29.78.131:32829 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 08 18:29:23 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:23 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:23.287426 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:23.281604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.129:32913)
0808 18:29:23.281877 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:23.281891 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:23.281974 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:23.282219 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:23.282221 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:23.282420 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:23.282518 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:23.282524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:23.283318 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:23.283320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:23.283872 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:23.283878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:23.284467 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:23.284471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:23.284554 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:23.285149 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:23.285170 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:23.287255 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":93,"threads_started":1}
W20260808 18:29:23.287590 7730 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:23.281647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:32913 (local address 127.29.78.190:42837)
0808 18:29:23.281911 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:23.281915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:23.281946 (+ 31us) server_negotiation.cc:407] Connection header received
0808 18:29:23.282066 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:23.282068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:23.282110 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:23.282183 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:23.282653 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:23.283221 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:23.284030 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:23.284327 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:23.284472 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:23.287450 (+ 2978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:32913: BlockingRecv error: recv got EOF from 127.29.78.129:32913 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":115,"threads_started":1}
Aug 08 18:29:23 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:23 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:23.478472 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:23.472655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.130:40295)
0808 18:29:23.473032 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:23.473050 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:23.473191 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:23.473476 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:23.473479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:23.473698 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:23.473787 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:23.473793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:23.474698 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:23.474704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:23.475287 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:23.475295 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:23.475763 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:23.475767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:23.475862 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:23.476429 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:23.476441 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:23.478250 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":113,"threads_started":1}
W20260808 18:29:23.478574 7730 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:23.472803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:40295 (local address 127.29.78.190:42837)
0808 18:29:23.472987 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:23.472992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:23.473131 (+ 139us) server_negotiation.cc:407] Connection header received
0808 18:29:23.473304 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:23.473307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:23.473365 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:23.473452 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:23.473917 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:23.474587 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:23.475416 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:23.475648 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:23.475759 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:23.478452 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:40295: BlockingRecv error: recv got EOF from 127.29.78.130:40295 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 08 18:29:23 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:23 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:23.667794 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:23.662250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.131:54095)
0808 18:29:23.662585 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:23.662598 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:23.662730 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:23.663013 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:23.663016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:23.663218 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:23.663311 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:23.663316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:23.664097 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:23.664102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:23.664680 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:23.664688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:23.665148 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:23.665152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:23.665230 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:23.665822 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:23.665834 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:23.667680 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":87,"threads_started":1}
W20260808 18:29:23.667963 7730 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:23.662338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:54095 (local address 127.29.78.190:42837)
0808 18:29:23.662517 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:23.662522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:23.662668 (+ 146us) server_negotiation.cc:407] Connection header received
0808 18:29:23.662827 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:23.662831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:23.662885 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:23.662971 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:23.663426 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:23.663982 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:23.664806 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:23.665034 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:23.665163 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:23.667809 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:54095: BlockingRecv error: recv got EOF from 127.29.78.131:54095 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 18:29:24 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:24 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:24.293866 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:24.288240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.129:51431)
0808 18:29:24.288555 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:24.288591 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:24.288709 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:24.288958 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:24.288960 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:24.289173 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:24.289276 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:24.289282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:24.290101 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:24.290104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:24.290775 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:24.290786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:24.291220 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:24.291223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:24.291290 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:24.291789 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:24.291801 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:24.293746 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":86,"threads_started":1}
W20260808 18:29:24.294080 7735 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:24.288401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:51431 (local address 127.29.78.190:42837)
0808 18:29:24.288670 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:24.288673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:24.288686 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:29:24.288783 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:24.288785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:24.288824 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:24.288898 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:24.289437 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:24.289977 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:24.290895 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:24.291117 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:24.291194 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:24.293922 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:51431: BlockingRecv error: recv got EOF from 127.29.78.129:51431 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":102,"threads_started":1}
Aug 08 18:29:24 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:24 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:24.484790 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:24.479276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.130:33545)
0808 18:29:24.479562 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:24.479576 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:24.479676 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:24.479924 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:24.479927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:24.480126 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:24.480221 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:24.480226 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:24.480949 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:24.480951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:24.481526 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:24.481534 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:24.482003 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:24.482006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:24.482101 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:24.482748 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:24.482760 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:24.484678 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":72,"threads_started":1}
W20260808 18:29:24.484966 7735 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:24.479432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:33545 (local address 127.29.78.190:42837)
0808 18:29:24.479614 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:24.479617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:24.479633 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:29:24.479768 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:24.479770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:24.479815 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:24.479883 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:24.480325 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:24.480861 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:24.481633 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:24.481872 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:24.481993 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:24.484809 (+ 2816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:33545: BlockingRecv error: recv got EOF from 127.29.78.130:33545 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 08 18:29:24 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:24 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:24.673853 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:24.668510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.131:33625)
0808 18:29:24.668791 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:24.668804 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:24.668898 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:24.669128 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:24.669130 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:24.669327 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:24.669421 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:24.669426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:24.670249 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:24.670252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:24.670835 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:24.670844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:24.671304 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:24.671306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:24.671366 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:24.671861 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:24.671872 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:24.673742 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":94,"threads_started":1}
W20260808 18:29:24.674021 7735 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:24.668610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:33625 (local address 127.29.78.190:42837)
0808 18:29:24.668742 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:24.668746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:24.668869 (+ 123us) server_negotiation.cc:407] Connection header received
0808 18:29:24.668976 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:24.668978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:24.669022 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:24.669091 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:24.669557 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:24.670142 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:24.670953 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:24.671202 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:24.671290 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:24.673890 (+ 2600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:33625: BlockingRecv error: recv got EOF from 127.29.78.131:33625 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 08 18:29:25 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:25 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:25.300736 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:25.294824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.129:60433)
0808 18:29:25.295157 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:25.295171 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:25.295268 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:25.295539 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:25.295542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:25.295728 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:25.295823 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:25.295829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:25.296718 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:25.296720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:25.297262 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:25.297269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:25.297810 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:25.297813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:25.297880 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:25.298386 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:25.298422 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:25.300623 (+ 2201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":109,"threads_started":1}
W20260808 18:29:25.300976 7739 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:25.294882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:60433 (local address 127.29.78.190:42837)
0808 18:29:25.295200 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:25.295203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:25.295221 (+ 18us) server_negotiation.cc:407] Connection header received
0808 18:29:25.295365 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:25.295367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:25.295412 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:25.295491 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:25.295965 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:25.296614 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:25.297416 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:25.297704 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:25.297796 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:25.300807 (+ 3011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:60433: BlockingRecv error: recv got EOF from 127.29.78.129:60433 (error 108)
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":126,"threads_started":1}
Aug 08 18:29:25 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:25 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:25.491122 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:25.485518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.130:46651)
0808 18:29:25.485832 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:25.485850 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:25.485995 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:25.486271 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:25.486274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:25.486537 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:25.486630 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:25.486636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:25.487409 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:25.487412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:25.487977 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:25.487985 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:25.488461 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:25.488464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:25.488557 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:25.489057 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:25.489068 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:25.491005 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":106,"threads_started":1}
W20260808 18:29:25.491303 7739 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:25.485673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:46651 (local address 127.29.78.190:42837)
0808 18:29:25.485834 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:25.485841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:25.485935 (+ 94us) server_negotiation.cc:407] Connection header received
0808 18:29:25.486102 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:25.486106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:25.486161 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:25.486248 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:25.486752 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:25.487285 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:25.488109 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:25.488346 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:25.488457 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:25.491143 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:46651: BlockingRecv error: recv got EOF from 127.29.78.130:46651 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 08 18:29:25 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:25 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
W20260808 18:29:25.681676 7739 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:25.674862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:43801 (local address 127.29.78.190:42837)
0808 18:29:25.675057 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:25.675073 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:25.675176 (+ 103us) server_negotiation.cc:407] Connection header received
0808 18:29:25.675301 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:25.675304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:25.675389 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:25.675559 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:25.675921 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:25.677122 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:25.678216 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:25.678576 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:25.678685 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:25.681584 (+ 2899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:43801: BlockingRecv error: recv got EOF from 127.29.78.131:43801 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 18:29:25.681689 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:25.674791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.131:43801)
0808 18:29:25.675088 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:25.675101 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:25.675201 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:25.675529 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:25.675531 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:25.675706 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:25.675798 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:25.675803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:25.677298 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:25.677301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:25.678074 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:25.678085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:25.678686 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:25.678690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:25.678775 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:25.679455 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:25.679470 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:25.681461 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":96,"threads_started":1}
Aug 08 18:29:26 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:26 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:26.307622 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:26.301581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.129:46791)
0808 18:29:26.302114 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:26.302129 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:26.302228 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:26.302455 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:26.302457 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:26.302648 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:26.302746 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:26.302751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:26.303533 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:26.303537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:26.304229 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:26.304240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:26.304699 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:26.304702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:26.304781 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:26.305294 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:26.305308 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:26.307434 (+ 2126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":160,"threads_started":1}
W20260808 18:29:26.307720 7743 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:26.301649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:46791 (local address 127.29.78.190:42837)
0808 18:29:26.302030 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:26.302035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:26.302201 (+ 166us) server_negotiation.cc:407] Connection header received
0808 18:29:26.302296 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:26.302298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:26.302343 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:26.302437 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:26.302847 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:26.303385 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:26.304352 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:26.304587 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:26.304666 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:26.307580 (+ 2914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:46791: BlockingRecv error: recv got EOF from 127.29.78.129:46791 (error 108)
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":120,"threads_started":1}
Aug 08 18:29:26 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:26 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:26.497320 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:26.491913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.130:51601)
0808 18:29:26.492170 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:26.492194 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:26.492293 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:26.492611 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:26.492614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:26.492805 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:26.492923 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:26.492929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:26.493738 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:26.493741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:26.494304 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:26.494311 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:26.494783 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:26.494785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:26.494858 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:26.495411 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:26.495422 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:26.497194 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":126,"threads_started":1}
W20260808 18:29:26.497519 7743 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:26.491999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:51601 (local address 127.29.78.190:42837)
0808 18:29:26.492154 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:26.492158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:26.492264 (+ 106us) server_negotiation.cc:407] Connection header received
0808 18:29:26.492382 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:26.492384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:26.492440 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:26.492518 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:26.493038 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:26.493633 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:26.494441 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:26.494679 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:26.494771 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:26.497333 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:51601: BlockingRecv error: recv got EOF from 127.29.78.130:51601 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 08 18:29:26 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:26 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:26.688271 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:26.682545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.131:35621)
0808 18:29:26.682827 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:26.682839 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:26.682976 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:26.683280 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:26.683283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:26.683505 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:26.683598 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:26.683603 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:26.684369 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:26.684375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:26.684961 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:26.684969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:26.685456 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:26.685459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:26.685547 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:26.686111 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:26.686123 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:26.688091 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":93,"threads_started":1}
W20260808 18:29:26.688318 7743 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:26.682568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:35621 (local address 127.29.78.190:42837)
0808 18:29:26.682734 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:26.682739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:26.682916 (+ 177us) server_negotiation.cc:407] Connection header received
0808 18:29:26.683108 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:26.683111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:26.683169 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:26.683258 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:26.683719 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:26.684257 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:26.685094 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:26.685341 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:26.685454 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:26.688211 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:35621: BlockingRecv error: recv got EOF from 127.29.78.131:35621 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 18:29:27 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:27 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:27.318943 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:27.308669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.129:46341)
0808 18:29:27.309095 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:27.309108 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:27.309196 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:27.309499 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:27.309503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:27.309722 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:27.309856 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:27.309864 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:27.310710 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:27.310714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:27.311289 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:27.311298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:27.311766 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:27.311769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:27.311858 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:27.316792 (+ 4934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:27.316804 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:27.318769 (+ 1965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":101,"threads_started":1}
W20260808 18:29:27.319064 7747 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:27.308722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:46341 (local address 127.29.78.190:42837)
0808 18:29:27.309057 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:27.309062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:27.309184 (+ 122us) server_negotiation.cc:407] Connection header received
0808 18:29:27.309316 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:27.309319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:27.309365 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:27.309475 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:27.309986 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:27.310600 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:27.311413 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:27.311654 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:27.311764 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:27.318905 (+ 7141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:46341: BlockingRecv error: recv got EOF from 127.29.78.129:46341 (error 108)
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":130,"threads_started":1}
Aug 08 18:29:27 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:29:27 dist-test-slave-gp6n krb5kdc[6590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:29:27.503438 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:27.498181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42837 (local address 127.29.78.130:34609)
0808 18:29:27.498536 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:27.498554 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:27.498682 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:27.498930 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:27.498932 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:27.499132 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:27.499226 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:27.499231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:27.499930 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:27.499933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:27.500482 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:27.500488 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:27.500881 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:27.500883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:27.500946 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:27.501498 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:27.501510 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:27.503325 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:42837: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":96,"threads_started":1}
W20260808 18:29:27.503692 7747 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:27.498226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:34609 (local address 127.29.78.190:42837)
0808 18:29:27.498375 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:27.498380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:27.498642 (+ 262us) server_negotiation.cc:407] Connection header received
0808 18:29:27.498777 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:27.498780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:27.498823 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:27.498890 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:27.499323 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:27.499841 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:27.500576 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:27.500789 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:27.500862 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:27.503570 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:34609: BlockingRecv error: recv got EOF from 127.29.78.130:34609 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 18:29:27.583503 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 7225
I20260808 18:29:27.593778 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 7391
I20260808 18:29:27.600657 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 7535
I20260808 18:29:27.607225 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 7148
2026-08-08T18:29:27Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18177 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-08T18:29:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:29:27Z Disabled control of system clock
I20260808 18:29:27.639812 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:41471
--webserver_interface=127.29.78.190
--webserver_port=0
--builtin_ntp_servers=127.29.78.148:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:41471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 18:29:27.749502 7752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:27.749768 7752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:27.749801 7752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:27.753783 7752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:29:27.753842 7752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:27.753858 7752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:27.753875 7752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:29:27.753890 7752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:29:27.758764 7752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:41471
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:41471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:27.759841 7752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:27.761245 7752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:27.770041 7757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:27.770079 7752 server_base.cc:1034] running on GCE node
W20260808 18:29:27.770128 7760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:27.770041 7758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:27.770967 7752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:27.772037 7752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:27.773197 7752 hybrid_clock.cc:648] HybridClock initialized: now 1786213767773166 us; error 45 us; skew 500 ppm
I20260808 18:29:27.775156 7752 webserver.cc:460] Webserver started at http://127.29.78.190:45579/ using document root <none> and password file <none>
I20260808 18:29:27.775686 7752 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:27.775743 7752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:27.775906 7752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:27.777858 7752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "a1bda6cb8d4b4166b2256e3a4f924284"
format_stamp: "Formatted at 2026-08-08 18:29:27 on dist-test-slave-gp6n"
server_key: "f260d5438f1b5b9ae9dd7c885a56eff4"
server_key_iv: "381e99e8b02132a201509e77d520c12d"
server_key_version: "encryptionkey@0"
I20260808 18:29:27.778375 7752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "a1bda6cb8d4b4166b2256e3a4f924284"
format_stamp: "Formatted at 2026-08-08 18:29:27 on dist-test-slave-gp6n"
server_key: "f260d5438f1b5b9ae9dd7c885a56eff4"
server_key_iv: "381e99e8b02132a201509e77d520c12d"
server_key_version: "encryptionkey@0"
I20260808 18:29:27.781944 7752 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:29:27.784278 7766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:27.785491 7752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:29:27.785610 7752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "a1bda6cb8d4b4166b2256e3a4f924284"
format_stamp: "Formatted at 2026-08-08 18:29:27 on dist-test-slave-gp6n"
server_key: "f260d5438f1b5b9ae9dd7c885a56eff4"
server_key_iv: "381e99e8b02132a201509e77d520c12d"
server_key_version: "encryptionkey@0"
I20260808 18:29:27.785717 7752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:27.800994 7752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:27.801631 7752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:27.801823 7752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:27.808862 7752 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:41471
I20260808 18:29:27.808873 7824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:41471 every 8 connection(s)
I20260808 18:29:27.809790 7752 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:29:27.812866 7825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:27.815801 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 7752
I20260808 18:29:27.815935 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:29:27.816229 30010 external_mini_cluster.cc:1428] Setting key d84aff69a53171b0c3f756a2707cc5de
I20260808 18:29:27.818657 7825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284: Bootstrap starting.
I20260808 18:29:27.820900 7825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:27.821807 7825 log.cc:824] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284: Log is configured to *not* fsync() on all Append() calls
I20260808 18:29:27.823837 7825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284: No bootstrap required, opened a new log
I20260808 18:29:27.825459 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:27.817670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51402 (local address 127.29.78.190:41471)
0808 18:29:27.818279 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:27.818294 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:27.818330 (+ 36us) server_negotiation.cc:407] Connection header received
0808 18:29:27.819047 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:27.819065 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:27.819445 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:27.819712 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:27.820045 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:27.820964 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:27.822179 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:27.822703 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:27.822873 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:27.823743 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:27.823748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:27.823760 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:29:27.824196 (+ 436us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:27.824250 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:29:27.824271 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:27.824379 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:27.824535 (+ 156us) server_negotiation.cc:299] Negotiation successful
0808 18:29:27.824783 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":134,"threads_started":1}
I20260808 18:29:27.827623 7825 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1bda6cb8d4b4166b2256e3a4f924284" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41471 } }
I20260808 18:29:27.827934 7825 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:27.827997 7825 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1bda6cb8d4b4166b2256e3a4f924284, State: Initialized, Role: FOLLOWER
I20260808 18:29:27.828398 7825 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1bda6cb8d4b4166b2256e3a4f924284" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41471 } }
I20260808 18:29:27.828543 7825 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:29:27.828605 7825 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:29:27.828722 7825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:27.829612 7825 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1bda6cb8d4b4166b2256e3a4f924284" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41471 } }
I20260808 18:29:27.829957 7825 leader_election.cc:302] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1bda6cb8d4b4166b2256e3a4f924284; no voters:
I20260808 18:29:27.830269 7825 leader_election.cc:288] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:29:27.830391 7830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:29:27.830668 7830 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284 [term 1 LEADER]: Becoming Leader. State: Replica: a1bda6cb8d4b4166b2256e3a4f924284, State: Running, Role: LEADER
I20260808 18:29:27.831014 7830 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1bda6cb8d4b4166b2256e3a4f924284" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41471 } }
I20260808 18:29:27.831509 7825 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:29:27.832727 7832 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1bda6cb8d4b4166b2256e3a4f924284. Latest consensus state: current_term: 1 leader_uuid: "a1bda6cb8d4b4166b2256e3a4f924284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1bda6cb8d4b4166b2256e3a4f924284" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41471 } } }
I20260808 18:29:27.832737 7831 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1bda6cb8d4b4166b2256e3a4f924284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1bda6cb8d4b4166b2256e3a4f924284" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41471 } } }
I20260808 18:29:27.832901 7832 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284 [sys.catalog]: This master's current role is: LEADER
I20260808 18:29:27.832942 7831 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284 [sys.catalog]: This master's current role is: LEADER
I20260808 18:29:27.833948 7841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:29:27.836503 7841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:29:27.841588 7841 catalog_manager.cc:1347] Generated new cluster ID: 07867ef8a2ad4a629d0833fda732efa4
I20260808 18:29:27.841655 7841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:29:27.854595 7841 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:29:27.855733 7841 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:29:27.869117 7841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1bda6cb8d4b4166b2256e3a4f924284: Generated new TSK 0
I20260808 18:29:27.869940 7841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:29:27.877923 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41471
--builtin_ntp_servers=127.29.78.148:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 18:29:27.989799 7850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:27.990031 7850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:27.990069 7850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:27.994061 7850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:27.994114 7850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:27.994199 7850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:29:27.998785 7850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:27.999780 7850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:28.001089 7850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:28.010609 7855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:28.010609 7856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:28.010612 7858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:28.010864 7850 server_base.cc:1034] running on GCE node
I20260808 18:29:28.011355 7850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:28.012017 7850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:28.013208 7850 hybrid_clock.cc:648] HybridClock initialized: now 1786213768013198 us; error 28 us; skew 500 ppm
I20260808 18:29:28.015055 7850 webserver.cc:460] Webserver started at http://127.29.78.129:34219/ using document root <none> and password file <none>
I20260808 18:29:28.015635 7850 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:28.015729 7850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:28.015960 7850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:28.017976 7850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "64319a91bd9a4f17b0e1e1b0b7834269"
format_stamp: "Formatted at 2026-08-08 18:29:28 on dist-test-slave-gp6n"
server_key: "bcc0f464f79d069a30642b718e58177a"
server_key_iv: "7328b720c9ff1b453f01a6eadff5fce2"
server_key_version: "encryptionkey@0"
I20260808 18:29:28.018611 7850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "64319a91bd9a4f17b0e1e1b0b7834269"
format_stamp: "Formatted at 2026-08-08 18:29:28 on dist-test-slave-gp6n"
server_key: "bcc0f464f79d069a30642b718e58177a"
server_key_iv: "7328b720c9ff1b453f01a6eadff5fce2"
server_key_version: "encryptionkey@0"
I20260808 18:29:28.022146 7850 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:29:28.024893 7864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:28.026050 7850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 18:29:28.026196 7850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "64319a91bd9a4f17b0e1e1b0b7834269"
format_stamp: "Formatted at 2026-08-08 18:29:28 on dist-test-slave-gp6n"
server_key: "bcc0f464f79d069a30642b718e58177a"
server_key_iv: "7328b720c9ff1b453f01a6eadff5fce2"
server_key_version: "encryptionkey@0"
I20260808 18:29:28.026314 7850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:28.051451 7850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:28.052181 7850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:28.052445 7850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:28.053102 7850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:29:28.054250 7850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:28.054301 7850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:28.054340 7850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:28.054369 7850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:28.063169 7850 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:38689
I20260808 18:29:28.063199 7991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:38689 every 8 connection(s)
I20260808 18:29:28.064119 7850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:29:28.065145 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 7850
I20260808 18:29:28.065258 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:29:28.065550 30010 external_mini_cluster.cc:1428] Setting key 96eade4eddb72cb01a4e015ba4723d50
I20260808 18:29:28.067713 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41471
--builtin_ntp_servers=127.29.78.148:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 18:29:28.074054 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:28.066038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:52787 (local address 127.29.78.190:41471)
0808 18:29:28.066179 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:28.066184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:28.067443 (+ 1259us) server_negotiation.cc:407] Connection header received
0808 18:29:28.068945 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:28.068948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:28.069006 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:28.069111 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:28.069999 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:28.070697 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:28.071840 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:28.072271 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:28.072405 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:28.073459 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:28.073466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:28.073472 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:29:28.073489 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:28.073515 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:29:28.073539 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:28.073629 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:28.073772 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 18:29:28.073848 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 18:29:28.074832 7994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:28.066549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:41471 (local address 127.29.78.129:52787)
0808 18:29:28.067303 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:28.067346 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:28.068728 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:28.069286 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:28.069292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:28.069386 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:28.069864 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:28.069876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:28.070822 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:28.070825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:28.071712 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:28.071721 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:28.072418 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:28.072421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:28.072569 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:28.073204 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:29:28.073205 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:28.073244 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:29:28.073303 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:28.073633 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:28.073640 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:28.073644 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 18:29:28.073739 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 18:29:28.073951 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":130,"threads_started":1}
I20260808 18:29:28.076323 7992 heartbeater.cc:347] Connected to a master server at 127.29.78.190:41471
I20260808 18:29:28.076539 7992 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:28.077028 7992 heartbeater.cc:511] Master 127.29.78.190:41471 requested a full tablet report, sending...
I20260808 18:29:28.078730 7785 ts_manager.cc:194] Registered new tserver with Master: 64319a91bd9a4f17b0e1e1b0b7834269 (127.29.78.129:38689)
I20260808 18:29:28.080333 7785 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.129:52787
W20260808 18:29:28.179716 7995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:28.179975 7995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:28.180011 7995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:28.183877 7995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:28.183923 7995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:28.183974 7995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:29:28.188498 7995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:28.189541 7995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:28.190871 7995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:28.200275 8003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:28.200284 8001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:28.200284 8000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:28.200547 7995 server_base.cc:1034] running on GCE node
I20260808 18:29:28.201049 7995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:28.201656 7995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:28.202811 7995 hybrid_clock.cc:648] HybridClock initialized: now 1786213768202797 us; error 26 us; skew 500 ppm
I20260808 18:29:28.204631 7995 webserver.cc:460] Webserver started at http://127.29.78.130:33729/ using document root <none> and password file <none>
I20260808 18:29:28.205194 7995 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:28.205256 7995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:28.205495 7995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:28.207588 7995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "0b63175a527e49a5a142b5b345405939"
format_stamp: "Formatted at 2026-08-08 18:29:28 on dist-test-slave-gp6n"
server_key: "725f4aefec501f17462340065804c08c"
server_key_iv: "0edfb1b2b180a7879950b23fb42f7730"
server_key_version: "encryptionkey@0"
I20260808 18:29:28.208148 7995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "0b63175a527e49a5a142b5b345405939"
format_stamp: "Formatted at 2026-08-08 18:29:28 on dist-test-slave-gp6n"
server_key: "725f4aefec501f17462340065804c08c"
server_key_iv: "0edfb1b2b180a7879950b23fb42f7730"
server_key_version: "encryptionkey@0"
I20260808 18:29:28.211767 7995 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:29:28.214190 8012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:28.215482 7995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:29:28.215651 7995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "0b63175a527e49a5a142b5b345405939"
format_stamp: "Formatted at 2026-08-08 18:29:28 on dist-test-slave-gp6n"
server_key: "725f4aefec501f17462340065804c08c"
server_key_iv: "0edfb1b2b180a7879950b23fb42f7730"
server_key_version: "encryptionkey@0"
I20260808 18:29:28.215746 7995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:28.243485 7995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:28.244158 7995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:28.244356 7995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:28.244935 7995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:29:28.245848 7995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:28.245895 7995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:28.245965 7995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:28.246002 7995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:28.254520 7995 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:32939
I20260808 18:29:28.254604 8138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:32939 every 8 connection(s)
I20260808 18:29:28.255429 7995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:29:28.263805 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:28.257201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:57815 (local address 127.29.78.190:41471)
0808 18:29:28.257334 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:28.257338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:28.258341 (+ 1003us) server_negotiation.cc:407] Connection header received
0808 18:29:28.259703 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:28.259706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:28.259752 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:28.259825 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:28.260603 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:28.261114 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:28.262177 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:28.262384 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:28.262477 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:28.263424 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:28.263427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:28.263429 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:29:28.263440 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:28.263456 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:29:28.263463 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:28.263530 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:28.263651 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 18:29:28.263697 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 18:29:28.264569 8141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:28.257574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:41471 (local address 127.29.78.130:57815)
0808 18:29:28.258199 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:28.258239 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:28.259519 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:28.259948 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:28.259954 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:28.260042 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:28.260486 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:28.260497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:28.261216 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:28.261218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:28.262074 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:28.262082 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:28.262536 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:28.262538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:28.262678 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:28.263197 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:29:28.263199 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:28.263235 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:29:28.263292 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:28.263534 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:28.263540 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:28.263545 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 18:29:28.263634 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 18:29:28.263826 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
I20260808 18:29:28.265062 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 7995
I20260808 18:29:28.265177 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:29:28.265583 30010 external_mini_cluster.cc:1428] Setting key 587560c5c67a353d6c096a2c722eeaa6
I20260808 18:29:28.265821 8139 heartbeater.cc:347] Connected to a master server at 127.29.78.190:41471
I20260808 18:29:28.266052 8139 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:28.266587 8139 heartbeater.cc:511] Master 127.29.78.190:41471 requested a full tablet report, sending...
I20260808 18:29:28.267571 7785 ts_manager.cc:194] Registered new tserver with Master: 0b63175a527e49a5a142b5b345405939 (127.29.78.130:32939)
I20260808 18:29:28.267661 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41471
--builtin_ntp_servers=127.29.78.148:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 18:29:28.268182 7785 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.130:57815
W20260808 18:29:28.379626 8142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:28.379889 8142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:28.379953 8142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:28.383908 8142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:28.383977 8142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:28.384058 8142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:29:28.388681 8142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:28.389714 8142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:28.391162 8142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:29:28.400676 8142 server_base.cc:1034] running on GCE node
W20260808 18:29:28.400650 8148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:28.400705 8150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:28.400856 8147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:28.401240 8142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:28.401880 8142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:28.403064 8142 hybrid_clock.cc:648] HybridClock initialized: now 1786213768403050 us; error 26 us; skew 500 ppm
I20260808 18:29:28.404824 8142 webserver.cc:460] Webserver started at http://127.29.78.131:41399/ using document root <none> and password file <none>
I20260808 18:29:28.405416 8142 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:28.405507 8142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:28.405751 8142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:28.407902 8142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "c1cbd9102ed24a4080d4862af55afb1c"
format_stamp: "Formatted at 2026-08-08 18:29:28 on dist-test-slave-gp6n"
server_key: "ac0b7820269dbcd892de215fe9871ee4"
server_key_iv: "7c3409eae3f5fb89d03b7eded7d55bda"
server_key_version: "encryptionkey@0"
I20260808 18:29:28.408478 8142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "c1cbd9102ed24a4080d4862af55afb1c"
format_stamp: "Formatted at 2026-08-08 18:29:28 on dist-test-slave-gp6n"
server_key: "ac0b7820269dbcd892de215fe9871ee4"
server_key_iv: "7c3409eae3f5fb89d03b7eded7d55bda"
server_key_version: "encryptionkey@0"
I20260808 18:29:28.412118 8142 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 18:29:28.414561 8156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:28.415745 8142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:29:28.415875 8142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "c1cbd9102ed24a4080d4862af55afb1c"
format_stamp: "Formatted at 2026-08-08 18:29:28 on dist-test-slave-gp6n"
server_key: "ac0b7820269dbcd892de215fe9871ee4"
server_key_iv: "7c3409eae3f5fb89d03b7eded7d55bda"
server_key_version: "encryptionkey@0"
I20260808 18:29:28.415987 8142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:28.434367 8142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:28.435065 8142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:28.435261 8142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:28.435827 8142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:29:28.436861 8142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:28.436931 8142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:28.437000 8142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:28.437054 8142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:28.445396 8142 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:36481
I20260808 18:29:28.445475 8282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:36481 every 8 connection(s)
I20260808 18:29:28.446321 8142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:29:28.454664 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 8142
I20260808 18:29:28.454752 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:29:28.455024 30010 external_mini_cluster.cc:1428] Setting key 8621520a0cb796f2b8f40b75c3ad34ce
I20260808 18:29:28.456128 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:28.448097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:45945 (local address 127.29.78.190:41471)
0808 18:29:28.448280 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:28.448284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:28.449626 (+ 1342us) server_negotiation.cc:407] Connection header received
0808 18:29:28.451498 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:28.451502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:28.451568 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:28.451644 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:28.452429 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:28.452955 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:28.454013 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:28.454225 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:28.454299 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:28.455487 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:28.455491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:28.455494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:29:28.455506 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:28.455544 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:29:28.455555 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:28.455636 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:28.455853 (+ 217us) server_negotiation.cc:299] Negotiation successful
0808 18:29:28.455918 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 18:29:28.457530 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:28.448619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:41471 (local address 127.29.78.131:45945)
0808 18:29:28.449461 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:28.449516 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:28.451265 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:28.451768 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:28.451775 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:28.451853 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:28.452313 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:28.452324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:28.453069 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:28.453072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:28.453914 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:28.453922 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:28.454348 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:28.454351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:28.454558 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:28.455171 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:29:28.455173 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:28.455212 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:29:28.455307 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:28.455649 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:28.455658 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:28.455664 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 18:29:28.456482 (+ 818us) client_negotiation.cc:240] Negotiation successful
0808 18:29:28.456750 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":144,"threads_started":1}
I20260808 18:29:28.458539 8283 heartbeater.cc:347] Connected to a master server at 127.29.78.190:41471
I20260808 18:29:28.458752 8283 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:28.459228 8283 heartbeater.cc:511] Master 127.29.78.190:41471 requested a full tablet report, sending...
I20260808 18:29:28.460153 7785 ts_manager.cc:194] Registered new tserver with Master: c1cbd9102ed24a4080d4862af55afb1c (127.29.78.131:36481)
I20260808 18:29:28.460652 7785 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.131:45945
I20260808 18:29:28.468762 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 18:29:28.471460 7828 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:28.470617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51418 (local address 127.29.78.190:41471)
0808 18:29:28.470756 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:28.470761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:28.470942 (+ 181us) server_negotiation.cc:407] Connection header received
0808 18:29:28.471022 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:28.471024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:28.471066 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:28.471124 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:28.471348 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51418: BlockingRecv error: recv got EOF from 127.0.0.1:51418 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:29:28.471578 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 7850
I20260808 18:29:28.483332 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 7995
I20260808 18:29:28.489579 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 8142
I20260808 18:29:28.498608 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 7752
2026-08-08T18:29:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (889 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-08T18:29:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:29:28Z Disabled control of system clock
I20260808 18:29:28.529353 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:41257
--webserver_interface=127.29.78.190
--webserver_port=0
--builtin_ntp_servers=127.29.78.148:41225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:41257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 18:29:28.637811 8298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:28.638111 8298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:28.638199 8298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:28.642199 8298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:29:28.642259 8298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:28.642275 8298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:28.642292 8298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:29:28.642307 8298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:29:28.647063 8298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:41225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:41257
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:41257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:28.648099 8298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:28.649469 8298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:28.657758 8303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:28.657819 8298 server_base.cc:1034] running on GCE node
W20260808 18:29:28.657771 8306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:28.657799 8304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:28.658619 8298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:28.659621 8298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:28.660779 8298 hybrid_clock.cc:648] HybridClock initialized: now 1786213768660767 us; error 23 us; skew 500 ppm
I20260808 18:29:28.662616 8298 webserver.cc:460] Webserver started at http://127.29.78.190:38771/ using document root <none> and password file <none>
I20260808 18:29:28.663157 8298 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:28.663218 8298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:28.663462 8298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:28.665421 8298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "b54959ce49bf4d13986fc287b9eb86c0"
format_stamp: "Formatted at 2026-08-08 18:29:28 on dist-test-slave-gp6n"
server_key: "5cda73d7f172d7586135dd908dd0312f"
server_key_iv: "a6549b9de87ba1e254076be3cdc92122"
server_key_version: "encryptionkey@0"
I20260808 18:29:28.665961 8298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "b54959ce49bf4d13986fc287b9eb86c0"
format_stamp: "Formatted at 2026-08-08 18:29:28 on dist-test-slave-gp6n"
server_key: "5cda73d7f172d7586135dd908dd0312f"
server_key_iv: "a6549b9de87ba1e254076be3cdc92122"
server_key_version: "encryptionkey@0"
I20260808 18:29:28.669440 8298 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 18:29:28.671733 8314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:28.672857 8298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:29:28.673000 8298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "b54959ce49bf4d13986fc287b9eb86c0"
format_stamp: "Formatted at 2026-08-08 18:29:28 on dist-test-slave-gp6n"
server_key: "5cda73d7f172d7586135dd908dd0312f"
server_key_iv: "a6549b9de87ba1e254076be3cdc92122"
server_key_version: "encryptionkey@0"
I20260808 18:29:28.673110 8298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:28.677429 8298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:28.677981 8298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:28.678155 8298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:28.684746 8298 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:41257
I20260808 18:29:28.684762 8372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:41257 every 8 connection(s)
I20260808 18:29:28.685583 8298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:29:28.688651 8373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:28.693831 8373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0: Bootstrap starting.
I20260808 18:29:28.694834 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 8298
I20260808 18:29:28.694964 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:29:28.695291 30010 external_mini_cluster.cc:1428] Setting key 76f059fddb58fd724b1ff7baa7fa1b05
I20260808 18:29:28.696682 8373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:28.697412 8373 log.cc:824] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0: Log is configured to *not* fsync() on all Append() calls
I20260808 18:29:28.699393 8373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0: No bootstrap required, opened a new log
I20260808 18:29:28.700294 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:28.696382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42982 (local address 127.29.78.190:41257)
0808 18:29:28.697011 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:28.697034 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:28.697067 (+ 33us) server_negotiation.cc:407] Connection header received
0808 18:29:28.697706 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:28.697724 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:28.697987 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:28.698310 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:28.699128 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:28.699133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:28.699144 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:29:28.699154 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:28.699208 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:29:28.699224 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:28.699342 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:28.699441 (+ 99us) server_negotiation.cc:299] Negotiation successful
0808 18:29:28.699670 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":125,"threads_started":1}
I20260808 18:29:28.703338 8373 raft_consensus.cc:348] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54959ce49bf4d13986fc287b9eb86c0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41257 } }
I20260808 18:29:28.703574 8373 raft_consensus.cc:374] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:28.703652 8373 raft_consensus.cc:729] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b54959ce49bf4d13986fc287b9eb86c0, State: Initialized, Role: FOLLOWER
I20260808 18:29:28.704051 8373 consensus_queue.cc:260] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54959ce49bf4d13986fc287b9eb86c0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41257 } }
I20260808 18:29:28.704205 8373 raft_consensus.cc:388] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:29:28.704289 8373 raft_consensus.cc:482] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:29:28.704407 8373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:28.705241 8373 raft_consensus.cc:504] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54959ce49bf4d13986fc287b9eb86c0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41257 } }
I20260808 18:29:28.705569 8373 leader_election.cc:302] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b54959ce49bf4d13986fc287b9eb86c0; no voters:
I20260808 18:29:28.705850 8373 leader_election.cc:288] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:29:28.705953 8379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:29:28.706207 8379 raft_consensus.cc:686] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0 [term 1 LEADER]: Becoming Leader. State: Replica: b54959ce49bf4d13986fc287b9eb86c0, State: Running, Role: LEADER
I20260808 18:29:28.706552 8379 consensus_queue.cc:237] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54959ce49bf4d13986fc287b9eb86c0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41257 } }
I20260808 18:29:28.707096 8373 sys_catalog.cc:564] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:29:28.708192 8380 sys_catalog.cc:455] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b54959ce49bf4d13986fc287b9eb86c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54959ce49bf4d13986fc287b9eb86c0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41257 } } }
I20260808 18:29:28.708319 8380 sys_catalog.cc:458] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0 [sys.catalog]: This master's current role is: LEADER
I20260808 18:29:28.708173 8381 sys_catalog.cc:455] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b54959ce49bf4d13986fc287b9eb86c0. Latest consensus state: current_term: 1 leader_uuid: "b54959ce49bf4d13986fc287b9eb86c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54959ce49bf4d13986fc287b9eb86c0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41257 } } }
I20260808 18:29:28.708633 8381 sys_catalog.cc:458] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0 [sys.catalog]: This master's current role is: LEADER
W20260808 18:29:28.711656 8396 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:29:28.711742 8396 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:29:28.711803 8389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:29:28.712528 8389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:29:28.717770 8389 catalog_manager.cc:1347] Generated new cluster ID: 08c0f30469134d51b8d36de20bf46db6
I20260808 18:29:28.717840 8389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:29:28.747037 8389 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:29:28.747393 8389 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:29:28.755064 8389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b54959ce49bf4d13986fc287b9eb86c0: Generated new TSK 0
I20260808 18:29:28.755749 8389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:29:28.763556 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41257
--builtin_ntp_servers=127.29.78.148:41225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 18:29:28.875623 8400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:28.875854 8400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:28.875890 8400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:28.879927 8400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:28.879972 8400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:28.880028 8400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:29:28.884584 8400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:41225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:28.885486 8400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:28.886772 8400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:28.896193 8409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:28.896221 8407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:28.896356 8406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:28.896535 8400 server_base.cc:1034] running on GCE node
I20260808 18:29:28.896891 8400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:28.897512 8400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:28.898696 8400 hybrid_clock.cc:648] HybridClock initialized: now 1786213768898684 us; error 27 us; skew 500 ppm
I20260808 18:29:28.900426 8400 webserver.cc:460] Webserver started at http://127.29.78.129:37735/ using document root <none> and password file <none>
I20260808 18:29:28.900974 8400 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:28.901069 8400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:28.901294 8400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:28.903306 8400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "5515c1b5a325413480966894f245a77e"
format_stamp: "Formatted at 2026-08-08 18:29:28 on dist-test-slave-gp6n"
server_key: "22b38a3cf15efe1bcf589482a6cd09be"
server_key_iv: "caa9c4484ab55e49856215127cc4a23f"
server_key_version: "encryptionkey@0"
I20260808 18:29:28.903867 8400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "5515c1b5a325413480966894f245a77e"
format_stamp: "Formatted at 2026-08-08 18:29:28 on dist-test-slave-gp6n"
server_key: "22b38a3cf15efe1bcf589482a6cd09be"
server_key_iv: "caa9c4484ab55e49856215127cc4a23f"
server_key_version: "encryptionkey@0"
I20260808 18:29:28.907317 8400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:29:28.909809 8415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:28.911003 8400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:29:28.911131 8400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "5515c1b5a325413480966894f245a77e"
format_stamp: "Formatted at 2026-08-08 18:29:28 on dist-test-slave-gp6n"
server_key: "22b38a3cf15efe1bcf589482a6cd09be"
server_key_iv: "caa9c4484ab55e49856215127cc4a23f"
server_key_version: "encryptionkey@0"
I20260808 18:29:28.911240 8400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:28.915535 8400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:28.916088 8400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:28.916270 8400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:28.916785 8400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:29:28.917635 8400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:28.917713 8400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:28.917788 8400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:28.917824 8400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:28.926800 8400 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:45341
I20260808 18:29:28.926829 8536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:45341 every 8 connection(s)
I20260808 18:29:28.927685 8400 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:29:28.930393 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 8400
I20260808 18:29:28.930552 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:29:28.930891 30010 external_mini_cluster.cc:1428] Setting key 0899a016db74d431e572bea88ce72394
I20260808 18:29:28.933211 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41257
--builtin_ntp_servers=127.29.78.148:41225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 18:29:28.934136 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:28.929477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:37239 (local address 127.29.78.190:41257)
0808 18:29:28.929634 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:28.929639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:28.930785 (+ 1146us) server_negotiation.cc:407] Connection header received
0808 18:29:28.932415 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:28.932418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:28.932451 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:28.932501 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:28.933673 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:28.933677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:28.933680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:29:28.933698 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:28.933717 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:29:28.933725 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:28.933817 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:28.933963 (+ 146us) server_negotiation.cc:299] Negotiation successful
0808 18:29:28.934007 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 18:29:28.935170 8541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:28.929868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:41257 (local address 127.29.78.129:37239)
0808 18:29:28.930632 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:28.930689 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:28.932206 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:28.932657 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:28.932664 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:28.932750 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:28.933398 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:29:28.933402 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:28.933456 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:29:28.933534 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:28.933807 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:28.933816 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:28.933822 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 18:29:28.934007 (+ 185us) client_negotiation.cc:240] Negotiation successful
0808 18:29:28.934251 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":113,"threads_started":1}
I20260808 18:29:28.936416 8537 heartbeater.cc:347] Connected to a master server at 127.29.78.190:41257
I20260808 18:29:28.936654 8537 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:28.937052 8537 heartbeater.cc:511] Master 127.29.78.190:41257 requested a full tablet report, sending...
I20260808 18:29:28.938709 8331 ts_manager.cc:194] Registered new tserver with Master: 5515c1b5a325413480966894f245a77e (127.29.78.129:45341)
W20260808 18:29:29.044101 8542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:29.044420 8542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:29.044497 8542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:29.048529 8542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:29.048622 8542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:29.048714 8542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:29:29.053447 8542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:41225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:29.054693 8542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:29.056093 8542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:29.065932 8550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:29.065948 8547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:29.065979 8548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:29.066282 8542 server_base.cc:1034] running on GCE node
I20260808 18:29:29.066731 8542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:29.067351 8542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:29.068513 8542 hybrid_clock.cc:648] HybridClock initialized: now 1786213769068497 us; error 28 us; skew 500 ppm
I20260808 18:29:29.070313 8542 webserver.cc:460] Webserver started at http://127.29.78.130:36799/ using document root <none> and password file <none>
I20260808 18:29:29.070907 8542 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:29.070968 8542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:29.071213 8542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:29.073230 8542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "65e172e9ffdb419085419f136848ca4a"
format_stamp: "Formatted at 2026-08-08 18:29:29 on dist-test-slave-gp6n"
server_key: "0b96fc0ac70e0ed611396b326b0d8633"
server_key_iv: "3192e15e9a45b0570591341fcdc16c49"
server_key_version: "encryptionkey@0"
I20260808 18:29:29.073798 8542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "65e172e9ffdb419085419f136848ca4a"
format_stamp: "Formatted at 2026-08-08 18:29:29 on dist-test-slave-gp6n"
server_key: "0b96fc0ac70e0ed611396b326b0d8633"
server_key_iv: "3192e15e9a45b0570591341fcdc16c49"
server_key_version: "encryptionkey@0"
I20260808 18:29:29.077587 8542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 18:29:29.080089 8557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:29.081233 8542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:29:29.081367 8542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "65e172e9ffdb419085419f136848ca4a"
format_stamp: "Formatted at 2026-08-08 18:29:29 on dist-test-slave-gp6n"
server_key: "0b96fc0ac70e0ed611396b326b0d8633"
server_key_iv: "3192e15e9a45b0570591341fcdc16c49"
server_key_version: "encryptionkey@0"
I20260808 18:29:29.081480 8542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:29.085916 8542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:29.086555 8542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:29.086732 8542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:29.087245 8542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:29:29.088110 8542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:29.088181 8542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:29.088253 8542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:29.088296 8542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:29.096911 8542 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:40307
I20260808 18:29:29.096961 8681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:40307 every 8 connection(s)
I20260808 18:29:29.097780 8542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:29:29.100283 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 8542
I20260808 18:29:29.100435 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:29:29.100772 30010 external_mini_cluster.cc:1428] Setting key 21bcd620ed2424fc3b1341184127ac19
I20260808 18:29:29.102895 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41257
--builtin_ntp_servers=127.29.78.148:41225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 18:29:29.104420 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:29.099575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:42449 (local address 127.29.78.190:41257)
0808 18:29:29.099716 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:29.099719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:29.100817 (+ 1098us) server_negotiation.cc:407] Connection header received
0808 18:29:29.102353 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:29.102357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:29.102431 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:29.102522 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:29.103816 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:29.103822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:29.103828 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:29:29.103845 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:29.103875 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:29:29.103889 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:29.103991 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:29.104164 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 18:29:29.104217 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 18:29:29.105206 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:29.099940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:41257 (local address 127.29.78.130:42449)
0808 18:29:29.100645 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:29.100696 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:29.102109 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:29.102715 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:29.102723 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:29.102836 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:29.103524 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:29:29.103526 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:29.103568 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:29:29.103651 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:29.104002 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:29.104010 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:29.104015 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 18:29:29.104174 (+ 159us) client_negotiation.cc:240] Negotiation successful
0808 18:29:29.104374 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":133,"threads_started":1}
I20260808 18:29:29.106477 8682 heartbeater.cc:347] Connected to a master server at 127.29.78.190:41257
I20260808 18:29:29.106722 8682 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:29.107141 8682 heartbeater.cc:511] Master 127.29.78.190:41257 requested a full tablet report, sending...
I20260808 18:29:29.108281 8331 ts_manager.cc:194] Registered new tserver with Master: 65e172e9ffdb419085419f136848ca4a (127.29.78.130:40307)
W20260808 18:29:29.215178 8685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:29.215417 8685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:29.215456 8685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:29.222084 8685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:29.222159 8685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:29.222263 8685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:29:29.230594 8685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:41225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:29.232489 8685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:29.234261 8685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:29.246569 8693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:29.246624 8691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:29.246624 8690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:29.246775 8685 server_base.cc:1034] running on GCE node
I20260808 18:29:29.247213 8685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:29.247815 8685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:29.248975 8685 hybrid_clock.cc:648] HybridClock initialized: now 1786213769248958 us; error 28 us; skew 500 ppm
I20260808 18:29:29.250845 8685 webserver.cc:460] Webserver started at http://127.29.78.131:37961/ using document root <none> and password file <none>
I20260808 18:29:29.251408 8685 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:29.251469 8685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:29.251718 8685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:29.253721 8685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "358dd263d08f4d3ba5d22bccfe8dbdfa"
format_stamp: "Formatted at 2026-08-08 18:29:29 on dist-test-slave-gp6n"
server_key: "fc0f4dbd4cbf99a4511c967559f6104a"
server_key_iv: "b2e435697eb4d6c69cecfeb3a289de65"
server_key_version: "encryptionkey@0"
I20260808 18:29:29.254288 8685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "358dd263d08f4d3ba5d22bccfe8dbdfa"
format_stamp: "Formatted at 2026-08-08 18:29:29 on dist-test-slave-gp6n"
server_key: "fc0f4dbd4cbf99a4511c967559f6104a"
server_key_iv: "b2e435697eb4d6c69cecfeb3a289de65"
server_key_version: "encryptionkey@0"
I20260808 18:29:29.257886 8685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 18:29:29.260236 8699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:29.261390 8685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:29:29.261503 8685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "358dd263d08f4d3ba5d22bccfe8dbdfa"
format_stamp: "Formatted at 2026-08-08 18:29:29 on dist-test-slave-gp6n"
server_key: "fc0f4dbd4cbf99a4511c967559f6104a"
server_key_iv: "b2e435697eb4d6c69cecfeb3a289de65"
server_key_version: "encryptionkey@0"
I20260808 18:29:29.261619 8685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:29.265993 8685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:29.266592 8685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:29.266721 8685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:29.267179 8685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:29:29.267943 8685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:29.268004 8685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:29.268047 8685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:29.268069 8685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:29.276235 8685 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:46809
I20260808 18:29:29.276301 8824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:46809 every 8 connection(s)
I20260808 18:29:29.277144 8685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:29:29.280721 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 8685
I20260808 18:29:29.280838 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:29:29.281199 30010 external_mini_cluster.cc:1428] Setting key d62567976695b38e7b36bc5f73dc3a60
I20260808 18:29:29.283600 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:29.278867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:37719 (local address 127.29.78.190:41257)
0808 18:29:29.279003 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:29.279007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:29.279965 (+ 958us) server_negotiation.cc:407] Connection header received
0808 18:29:29.281566 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:29.281570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:29.281619 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:29.281711 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:29.283079 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:29.283083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:29.283085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:29:29.283097 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:29.283121 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:29:29.283136 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:29.283236 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:29.283412 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 18:29:29.283459 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 18:29:29.284428 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:29.279235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:41257 (local address 127.29.78.131:37719)
0808 18:29:29.279831 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:29.279868 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:29.281327 (+ 1459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:29.282070 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:29.282076 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:29.282184 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:29.282817 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:29:29.282819 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:29.282857 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:29:29.282919 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:29.283251 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:29.283259 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:29.283264 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 18:29:29.283431 (+ 167us) client_negotiation.cc:240] Negotiation successful
0808 18:29:29.283656 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":105,"threads_started":1}
I20260808 18:29:29.285456 8825 heartbeater.cc:347] Connected to a master server at 127.29.78.190:41257
I20260808 18:29:29.285642 8825 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:29.285970 8825 heartbeater.cc:511] Master 127.29.78.190:41257 requested a full tablet report, sending...
I20260808 18:29:29.286883 8331 ts_manager.cc:194] Registered new tserver with Master: 358dd263d08f4d3ba5d22bccfe8dbdfa (127.29.78.131:46809)
I20260808 18:29:29.295038 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 18:29:29.297781 8377 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:29:29.296855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42990 (local address 127.29.78.190:41257)
0808 18:29:29.296988 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:29.296992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:29.297177 (+ 185us) server_negotiation.cc:407] Connection header received
0808 18:29:29.297331 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:29.297333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:29.297364 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:29.297445 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:29.297667 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42990: BlockingRecv error: recv got EOF from 127.0.0.1:42990 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 18:29:29.297994 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 8400
I20260808 18:29:29.306769 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 8542
I20260808 18:29:29.317610 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 8685
I20260808 18:29:29.323887 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 8298
2026-08-08T18:29:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (825 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:29:29 dist-test-slave-gp6n krb5kdc[8836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:29:29 dist-test-slave-gp6n krb5kdc[8836](info): set up 2 sockets
Aug 08 18:29:29 dist-test-slave-gp6n krb5kdc[8836](info): commencing operation
krb5kdc: starting...
W20260808 18:29:30.520272 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:29:30 dist-test-slave-gp6n krb5kdc[8836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:29:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:29:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:29:30.646710 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:33977
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:33977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:29:30.757333 8849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:30.757637 8849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:30.757719 8849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:30.761737 8849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:29:30.761837 8849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:30.761883 8849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:30.761940 8849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:29:30.761979 8849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:29:30.767243 8849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:33977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:33977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:30.768293 8849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:30.769301 8849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:30.777762 8854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:30.777797 8857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:30.777768 8855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:30.777998 8849 server_base.cc:1034] running on GCE node
I20260808 18:29:30.778720 8849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:30.779764 8849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:30.780952 8849 hybrid_clock.cc:648] HybridClock initialized: now 1786213770780940 us; error 37 us; skew 500 ppm
Aug 08 18:29:30 dist-test-slave-gp6n krb5kdc[8836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213770, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:30.783849 8849 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:29:30.784669 8849 webserver.cc:460] Webserver started at http://127.29.78.190:43699/ using document root <none> and password file <none>
I20260808 18:29:30.785202 8849 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:30.785262 8849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:30.785485 8849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:30.787849 8849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "a8c4af6719dc4dedbcf671cd2f8581ff"
format_stamp: "Formatted at 2026-08-08 18:29:30 on dist-test-slave-gp6n"
server_key: "b6eed7c31868e4dd3ba71b721bb3757c"
server_key_iv: "45f9673390339734330e0acb9988e2ae"
server_key_version: "encryptionkey@0"
I20260808 18:29:30.788425 8849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "a8c4af6719dc4dedbcf671cd2f8581ff"
format_stamp: "Formatted at 2026-08-08 18:29:30 on dist-test-slave-gp6n"
server_key: "b6eed7c31868e4dd3ba71b721bb3757c"
server_key_iv: "45f9673390339734330e0acb9988e2ae"
server_key_version: "encryptionkey@0"
I20260808 18:29:30.791947 8849 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:29:30.794209 8865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:30.795397 8849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:29:30.795519 8849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "a8c4af6719dc4dedbcf671cd2f8581ff"
format_stamp: "Formatted at 2026-08-08 18:29:30 on dist-test-slave-gp6n"
server_key: "b6eed7c31868e4dd3ba71b721bb3757c"
server_key_iv: "45f9673390339734330e0acb9988e2ae"
server_key_version: "encryptionkey@0"
I20260808 18:29:30.795626 8849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:30.810308 8849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:30.813172 8849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:30.813369 8849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:30.820330 8849 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:33977
I20260808 18:29:30.820363 8921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:33977 every 8 connection(s)
I20260808 18:29:30.821274 8849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:29:30.823019 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 8849
I20260808 18:29:30.823117 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:29:30.823410 30010 external_mini_cluster.cc:1428] Setting key 9cc4fde93242cef7118d315831995f56
I20260808 18:29:30.824286 8922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:30.830734 8922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff: Bootstrap starting.
Aug 08 18:29:30 dist-test-slave-gp6n krb5kdc[8836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:29:30.833194 8922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:30.833917 8922 log.cc:824] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff: Log is configured to *not* fsync() on all Append() calls
I20260808 18:29:30.835772 8922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff: No bootstrap required, opened a new log
I20260808 18:29:30.837435 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:30.824743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42924 (local address 127.29.78.190:33977)
0808 18:29:30.825144 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:30.825159 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:30.825204 (+ 45us) server_negotiation.cc:407] Connection header received
0808 18:29:30.825849 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:30.825866 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:30.826180 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:30.826506 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:30.827402 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:30.828364 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:30.829579 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:30.830032 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:30.830150 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:30.832818 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:30.832823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:30.832834 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:30.832859 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:30.834733 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:30.835171 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:30.835181 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:30.835185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:30.835290 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:30.835582 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:30.835586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:30.835588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:30.836015 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:30.836208 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:30.836493 (+ 285us) server_negotiation.cc:299] Negotiation successful
0808 18:29:30.836766 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":133,"threads_started":1}
I20260808 18:29:30.839305 8922 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c4af6719dc4dedbcf671cd2f8581ff" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33977 } }
I20260808 18:29:30.839609 8922 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:30.839669 8922 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c4af6719dc4dedbcf671cd2f8581ff, State: Initialized, Role: FOLLOWER
I20260808 18:29:30.840019 8922 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c4af6719dc4dedbcf671cd2f8581ff" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33977 } }
I20260808 18:29:30.840132 8922 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:29:30.840170 8922 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:29:30.840232 8922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:30.841024 8922 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c4af6719dc4dedbcf671cd2f8581ff" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33977 } }
I20260808 18:29:30.841305 8922 leader_election.cc:302] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8c4af6719dc4dedbcf671cd2f8581ff; no voters:
I20260808 18:29:30.841549 8922 leader_election.cc:288] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:29:30.841697 8927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:29:30.841964 8927 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff [term 1 LEADER]: Becoming Leader. State: Replica: a8c4af6719dc4dedbcf671cd2f8581ff, State: Running, Role: LEADER
I20260808 18:29:30.842262 8927 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c4af6719dc4dedbcf671cd2f8581ff" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33977 } }
I20260808 18:29:30.842787 8922 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:29:30.843979 8928 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8c4af6719dc4dedbcf671cd2f8581ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c4af6719dc4dedbcf671cd2f8581ff" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33977 } } }
I20260808 18:29:30.844134 8928 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff [sys.catalog]: This master's current role is: LEADER
I20260808 18:29:30.844437 8929 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8c4af6719dc4dedbcf671cd2f8581ff. Latest consensus state: current_term: 1 leader_uuid: "a8c4af6719dc4dedbcf671cd2f8581ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c4af6719dc4dedbcf671cd2f8581ff" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33977 } } }
I20260808 18:29:30.844559 8929 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff [sys.catalog]: This master's current role is: LEADER
I20260808 18:29:30.844501 8936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:29:30.847608 8936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:29:30.852931 8936 catalog_manager.cc:1347] Generated new cluster ID: 62aa8ac92fba4c44b07db78f933b625d
I20260808 18:29:30.853013 8936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:29:30.876917 8936 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:29:30.877899 8936 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:29:30.884253 8936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff: Generated new TSK 0
I20260808 18:29:30.884924 8936 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260808 18:29:30.918255 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:33977
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:29:31.028930 8948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:31.029202 8948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:31.029244 8948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:31.033526 8948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:31.033599 8948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:31.033694 8948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:29:31.039101 8948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:33977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:31.040155 8948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:31.041190 8948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:31.050877 8954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:31.050905 8956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:31.050974 8953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:31.051115 8948 server_base.cc:1034] running on GCE node
I20260808 18:29:31.051573 8948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:31.052174 8948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:31.053366 8948 hybrid_clock.cc:648] HybridClock initialized: now 1786213771053350 us; error 32 us; skew 500 ppm
Aug 08 18:29:31 dist-test-slave-gp6n krb5kdc[8836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213771, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:31.056347 8948 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:29:31.057168 8948 webserver.cc:460] Webserver started at http://127.29.78.129:34275/ using document root <none> and password file <none>
I20260808 18:29:31.057693 8948 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:31.057752 8948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:31.057981 8948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:31.059990 8948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "7fd4b95bd5fa48b3a647f21e63eeaee7"
format_stamp: "Formatted at 2026-08-08 18:29:31 on dist-test-slave-gp6n"
server_key: "c3126eeb905e4548d229281676195963"
server_key_iv: "9476254b0d73706a7b5c203073b56d15"
server_key_version: "encryptionkey@0"
I20260808 18:29:31.060557 8948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "7fd4b95bd5fa48b3a647f21e63eeaee7"
format_stamp: "Formatted at 2026-08-08 18:29:31 on dist-test-slave-gp6n"
server_key: "c3126eeb905e4548d229281676195963"
server_key_iv: "9476254b0d73706a7b5c203073b56d15"
server_key_version: "encryptionkey@0"
I20260808 18:29:31.064157 8948 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260808 18:29:31.066707 8964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:31.067858 8948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260808 18:29:31.067981 8948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "7fd4b95bd5fa48b3a647f21e63eeaee7"
format_stamp: "Formatted at 2026-08-08 18:29:31 on dist-test-slave-gp6n"
server_key: "c3126eeb905e4548d229281676195963"
server_key_iv: "9476254b0d73706a7b5c203073b56d15"
server_key_version: "encryptionkey@0"
I20260808 18:29:31.068089 8948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:31.084604 8948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:31.087940 8948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:31.088157 8948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:31.088770 8948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:29:31.089625 8948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:31.089695 8948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:31.089761 8948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:31.089804 8948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:31.098809 8948 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:39299
I20260808 18:29:31.098858 9084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:39299 every 8 connection(s)
I20260808 18:29:31.099815 8948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:29:31.105261 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 8948
I20260808 18:29:31.105396 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:29:31.105679 30010 external_mini_cluster.cc:1428] Setting key e93844c1ba746f62f803023c5c337349
Aug 08 18:29:31 dist-test-slave-gp6n krb5kdc[8836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213771, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:29:31.115489 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:31.101774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:35517 (local address 127.29.78.190:33977)
0808 18:29:31.101924 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:31.101928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:31.102811 (+ 883us) server_negotiation.cc:407] Connection header received
0808 18:29:31.104260 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:31.104264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:31.104323 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:31.104429 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:31.105726 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.106495 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.108200 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.108434 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.108526 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:31.111183 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:31.111190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:31.111195 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:31.111227 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:31.113182 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:31.113718 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:31.113724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:31.113727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:31.113836 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:31.114225 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:31.114232 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:31.114236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:31.114519 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:31.114629 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:31.115113 (+ 484us) server_negotiation.cc:299] Negotiation successful
0808 18:29:31.115266 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 18:29:31.116155 9087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:31.102122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:33977 (local address 127.29.78.129:35517)
0808 18:29:31.102654 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:31.102699 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:31.104050 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:31.104607 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:31.104614 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:31.104979 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:31.105583 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:31.105599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.106623 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.106626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:31.108062 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:31.108077 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.108615 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.108622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:31.108859 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:31.109513 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:31.109534 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:31.111018 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:31.113319 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:31.113323 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:31.113335 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:31.113613 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:31.113954 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:31.113957 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:31.113958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:31.114120 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:31.114638 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:31.114644 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:31.114866 (+ 222us) client_negotiation.cc:769] Sending connection context
0808 18:29:31.115022 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 18:29:31.115267 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260808 18:29:31.117413 9085 heartbeater.cc:347] Connected to a master server at 127.29.78.190:33977
I20260808 18:29:31.117661 9085 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:31.118222 9085 heartbeater.cc:511] Master 127.29.78.190:33977 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
I20260808 18:29:31.119831 8882 ts_manager.cc:194] Registered new tserver with Master: 7fd4b95bd5fa48b3a647f21e63eeaee7 (127.29.78.129:39299)
I20260808 18:29:31.121475 8882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:35517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260808 18:29:31.137148 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:33977
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:29:31.261507 9090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:31.261776 9090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:31.261850 9090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:31.265834 9090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:31.265920 9090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:31.266012 9090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:29:31.271257 9090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:33977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:31.272363 9090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:31.273378 9090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:31.282972 9095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:31.283003 9098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:31.283167 9090 server_base.cc:1034] running on GCE node
W20260808 18:29:31.282984 9096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:31.283830 9090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:31.284448 9090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:31.285640 9090 hybrid_clock.cc:648] HybridClock initialized: now 1786213771285625 us; error 30 us; skew 500 ppm
Aug 08 18:29:31 dist-test-slave-gp6n krb5kdc[8836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213771, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:31.288852 9090 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:29:31.289674 9090 webserver.cc:460] Webserver started at http://127.29.78.130:35311/ using document root <none> and password file <none>
I20260808 18:29:31.290189 9090 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:31.290246 9090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:31.290460 9090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:31.292539 9090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "8249e5b58bf44ed1ae5b85891a582340"
format_stamp: "Formatted at 2026-08-08 18:29:31 on dist-test-slave-gp6n"
server_key: "d1e2e7bd3472a5d98ba57be22cbe168e"
server_key_iv: "ea43dcf3a9af28f43ac128360998e062"
server_key_version: "encryptionkey@0"
I20260808 18:29:31.293146 9090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "8249e5b58bf44ed1ae5b85891a582340"
format_stamp: "Formatted at 2026-08-08 18:29:31 on dist-test-slave-gp6n"
server_key: "d1e2e7bd3472a5d98ba57be22cbe168e"
server_key_iv: "ea43dcf3a9af28f43ac128360998e062"
server_key_version: "encryptionkey@0"
I20260808 18:29:31.296903 9090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260808 18:29:31.299397 9105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:31.300688 9090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:29:31.300808 9090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "8249e5b58bf44ed1ae5b85891a582340"
format_stamp: "Formatted at 2026-08-08 18:29:31 on dist-test-slave-gp6n"
server_key: "d1e2e7bd3472a5d98ba57be22cbe168e"
server_key_iv: "ea43dcf3a9af28f43ac128360998e062"
server_key_version: "encryptionkey@0"
I20260808 18:29:31.300917 9090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:31.326249 9090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:31.329674 9090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:31.329886 9090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:31.330570 9090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:29:31.331393 9090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:31.331441 9090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:31.331481 9090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:31.331534 9090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:31.340615 9090 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:41509
I20260808 18:29:31.340641 9232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:41509 every 8 connection(s)
I20260808 18:29:31.341536 9090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:29:31.344765 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 9090
I20260808 18:29:31.344887 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:29:31.345176 30010 external_mini_cluster.cc:1428] Setting key fbc8cd971e588ff3a18f51c806943ca4
Aug 08 18:29:31 dist-test-slave-gp6n krb5kdc[8836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213771, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260808 18:29:31.356835 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:31.343596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:45937 (local address 127.29.78.190:33977)
0808 18:29:31.343885 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:31.343890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:31.344726 (+ 836us) server_negotiation.cc:407] Connection header received
0808 18:29:31.346651 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:31.346654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:31.346708 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:31.346791 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:31.347956 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.348494 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.349663 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.349901 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.350015 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:31.352842 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:31.352853 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:31.352872 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:31.352909 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:31.354651 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:31.355160 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:31.355164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:31.355167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:31.355269 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:31.355618 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:31.355623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:31.355627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:31.355897 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:31.356022 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:31.356474 (+ 452us) server_negotiation.cc:299] Negotiation successful
0808 18:29:31.356625 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260808 18:29:31.357697 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:31.344060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:33977 (local address 127.29.78.130:45937)
0808 18:29:31.344514 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:31.344614 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:31.346326 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:31.346980 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:31.346987 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:31.347349 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:31.347813 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:31.347826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.348620 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.348624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:31.349529 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:31.349538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.350053 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.350057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:31.350228 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:31.350899 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:31.350928 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:31.352666 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:31.354773 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:31.354777 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:31.354789 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:31.355054 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:31.355367 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:31.355370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:31.355371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:31.355514 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:31.356011 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:31.356016 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:31.356233 (+ 217us) client_negotiation.cc:769] Sending connection context
0808 18:29:31.356390 (+ 157us) client_negotiation.cc:240] Negotiation successful
0808 18:29:31.356725 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":114,"threads_started":1}
I20260808 18:29:31.359061 9233 heartbeater.cc:347] Connected to a master server at 127.29.78.190:33977
I20260808 18:29:31.359282 9233 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
I20260808 18:29:31.359845 9233 heartbeater.cc:511] Master 127.29.78.190:33977 requested a full tablet report, sending...
I20260808 18:29:31.361014 8882 ts_manager.cc:194] Registered new tserver with Master: 8249e5b58bf44ed1ae5b85891a582340 (127.29.78.130:41509)
I20260808 18:29:31.361608 8882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:45937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260808 18:29:31.377326 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:33977
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:29:31.489019 9238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:29:31.489257 9238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:29:31.489351 9238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:29:31.493340 9238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:29:31.493393 9238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:29:31.493508 9238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:29:31.498555 9238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:33977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:29:31.499578 9238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:29:31.500594 9238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:29:31.509977 9248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:29:31.509981 9246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:31.510167 9238 server_base.cc:1034] running on GCE node
W20260808 18:29:31.510092 9245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:29:31.510676 9238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:29:31.511313 9238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:29:31.512501 9238 hybrid_clock.cc:648] HybridClock initialized: now 1786213771512480 us; error 31 us; skew 500 ppm
Aug 08 18:29:31 dist-test-slave-gp6n krb5kdc[8836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213771, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:29:31.515286 9238 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:29:31.516085 9238 webserver.cc:460] Webserver started at http://127.29.78.131:44089/ using document root <none> and password file <none>
I20260808 18:29:31.516619 9238 fs_manager.cc:359] Metadata directory not provided
I20260808 18:29:31.516706 9238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:29:31.516935 9238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:29:31.519250 9238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "db4c6bed85f54abf9d81b0d8657a7831"
format_stamp: "Formatted at 2026-08-08 18:29:31 on dist-test-slave-gp6n"
server_key: "bb2483a68628fd433916b55b5b755d1e"
server_key_iv: "b3d1fb265ce4a39e589c3bd980776d76"
server_key_version: "encryptionkey@0"
I20260808 18:29:31.520004 9238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "db4c6bed85f54abf9d81b0d8657a7831"
format_stamp: "Formatted at 2026-08-08 18:29:31 on dist-test-slave-gp6n"
server_key: "bb2483a68628fd433916b55b5b755d1e"
server_key_iv: "b3d1fb265ce4a39e589c3bd980776d76"
server_key_version: "encryptionkey@0"
I20260808 18:29:31.524763 9238 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:29:31.527482 9255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:31.528681 9238 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:29:31.528772 9238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "db4c6bed85f54abf9d81b0d8657a7831"
format_stamp: "Formatted at 2026-08-08 18:29:31 on dist-test-slave-gp6n"
server_key: "bb2483a68628fd433916b55b5b755d1e"
server_key_iv: "b3d1fb265ce4a39e589c3bd980776d76"
server_key_version: "encryptionkey@0"
I20260808 18:29:31.528849 9238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:29:31.548314 9238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:29:31.551051 9238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:29:31.551194 9238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:29:31.551685 9238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:29:31.552415 9238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:29:31.552460 9238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:31.552498 9238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:29:31.552520 9238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:29:31.560750 9238 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:35775
I20260808 18:29:31.560823 9381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:35775 every 8 connection(s)
I20260808 18:29:31.561607 9238 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:29:31.563692 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 9238
I20260808 18:29:31.563822 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:29:31.564124 30010 external_mini_cluster.cc:1428] Setting key 910ea98cac02d769133c9f71715f7734
Aug 08 18:29:31 dist-test-slave-gp6n krb5kdc[8836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213771, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:29:31.575229 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:31.563338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:35297 (local address 127.29.78.190:33977)
0808 18:29:31.563502 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:31.563507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:31.564400 (+ 893us) server_negotiation.cc:407] Connection header received
0808 18:29:31.566043 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:31.566047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:31.566110 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:31.566195 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:31.567514 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.568054 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.569132 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.569360 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.569448 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:31.571866 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:31.571869 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:31.571871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:31.571901 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:31.573404 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:31.573842 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:31.573845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:31.573846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:31.573928 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:31.574213 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:31.574216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:31.574217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:31.574384 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:31.574524 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:31.574985 (+ 461us) server_negotiation.cc:299] Negotiation successful
0808 18:29:31.575104 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 18:29:31.575920 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:31.563703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:33977 (local address 127.29.78.131:35297)
0808 18:29:31.564227 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:31.564278 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:31.565791 (+ 1513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:31.566458 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:31.566467 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:31.566903 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:29:31.567371 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:31.567384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.568159 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.568161 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:31.569022 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:31.569031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.569492 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.569495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:31.569651 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:31.570195 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:29:31.570211 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:29:31.571723 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:29:31.573499 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:31.573503 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:31.573513 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:31.573759 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:31.574003 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:29:31.574005 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:29:31.574006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:29:31.574131 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:29:31.574539 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:29:31.574544 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:29:31.574746 (+ 202us) client_negotiation.cc:769] Sending connection context
0808 18:29:31.574913 (+ 167us) client_negotiation.cc:240] Negotiation successful
0808 18:29:31.575142 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":173,"threads_started":1}
I20260808 18:29:31.577096 9382 heartbeater.cc:347] Connected to a master server at 127.29.78.190:33977
I20260808 18:29:31.577314 9382 heartbeater.cc:464] Registering TS with master...
I20260808 18:29:31.577854 9382 heartbeater.cc:511] Master 127.29.78.190:33977 requested a full tablet report, sending...
I20260808 18:29:31.578897 8882 ts_manager.cc:194] Registered new tserver with Master: db4c6bed85f54abf9d81b0d8657a7831 (127.29.78.131:35775)
I20260808 18:29:31.579444 8882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:35297
I20260808 18:29:31.589457 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:29:31.598775 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:31.591388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37166 (local address 127.29.78.190:33977)
0808 18:29:31.591521 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:31.591534 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:31.591709 (+ 175us) server_negotiation.cc:407] Connection header received
0808 18:29:31.591856 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:31.591861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:31.591904 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:31.591998 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:29:31.592789 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.593301 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.594139 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.594371 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.594548 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:31.595804 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:29:31.595808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:29:31.595811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:29:31.595835 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:29:31.597027 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:31.597455 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:31.597459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:31.597461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:31.597553 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:29:31.597833 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:29:31.597836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:29:31.597838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:29:31.598061 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:29:31.598195 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:31.598493 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 18:29:31.598617 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 18:29:31.601647 8882 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:29:31.604638 8882 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:29:31.618839 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:31.613108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:39299 (local address 127.0.0.1:36396)
0808 18:29:31.613692 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:31.613742 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:31.613862 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:31.614941 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:31.614947 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:31.614968 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:31.615124 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:31.615136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.616237 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.616243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:31.617437 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:31.617446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.618284 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.618289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:31.618430 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:31.618476 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 18:29:31.618587 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 18:29:31.618692 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":4,"thread_start_us":72,"threads_started":1}
I20260808 18:29:31.618866 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:31.613233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36396 (local address 127.29.78.129:39299)
0808 18:29:31.614425 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:31.614441 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:31.614464 (+ 23us) server_negotiation.cc:407] Connection header received
0808 18:29:31.614545 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:31.614554 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:31.614769 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:31.614938 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:31.615268 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.616104 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.617567 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.618161 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.618275 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:31.618445 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:31.618632 (+ 187us) server_negotiation.cc:299] Negotiation successful
0808 18:29:31.618724 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1058,"thread_start_us":89,"threads_started":1}
I20260808 18:29:31.619892 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:31.614214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59226 (local address 127.29.78.130:41509)
0808 18:29:31.614706 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:31.614716 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:31.614737 (+ 21us) server_negotiation.cc:407] Connection header received
0808 18:29:31.614810 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:31.614817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:31.614983 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:31.615120 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:31.615402 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.616569 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.618200 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.619164 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.619251 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:31.619372 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:31.619662 (+ 290us) server_negotiation.cc:299] Negotiation successful
0808 18:29:31.619746 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":120,"threads_started":1}
I20260808 18:29:31.620078 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:31.613832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57480 (local address 127.29.78.131:35775)
0808 18:29:31.614203 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:31.614215 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:31.614354 (+ 139us) server_negotiation.cc:407] Connection header received
0808 18:29:31.614623 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:31.614629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:31.614840 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:31.614986 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:31.615460 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.616569 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.618171 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.619175 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.619277 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:31.619401 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:31.619729 (+ 328us) server_negotiation.cc:299] Negotiation successful
0808 18:29:31.619825 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":139,"threads_started":1}
I20260808 18:29:31.620491 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:31.613158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:35775 (local address 127.0.0.1:57480)
0808 18:29:31.614253 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:31.614268 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:31.614456 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:31.615182 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:31.615187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:31.615206 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:31.615315 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:31.615324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.616730 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.616734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:31.618034 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:31.618046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.619465 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.619469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:31.619569 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:31.619591 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 18:29:31.620259 (+ 668us) client_negotiation.cc:240] Negotiation successful
0808 18:29:31.620341 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"mutex_wait_us":58,"thread_start_us":108,"threads_started":1}
I20260808 18:29:31.621996 9015 tablet_service.cc:1467] Processing CreateTablet for tablet 3c153e56492e41299f80bfa4116b124d (DEFAULT_TABLE table=test-table [id=b18b713be14b4ea284406340029bd0bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:29:31.622644 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:31.613784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:41509 (local address 127.0.0.1:59226)
0808 18:29:31.614629 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:31.614641 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:31.614713 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:31.615125 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:31.615128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:31.615145 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:31.615267 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:31.615276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.616730 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.616734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:31.618044 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:31.618051 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.619445 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.619447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:31.619520 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:31.619537 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 18:29:31.619630 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 18:29:31.619697 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":163,"thread_start_us":114,"threads_started":1}
I20260808 18:29:31.623142 9015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c153e56492e41299f80bfa4116b124d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:31.623275 9309 tablet_service.cc:1467] Processing CreateTablet for tablet 3c153e56492e41299f80bfa4116b124d (DEFAULT_TABLE table=test-table [id=b18b713be14b4ea284406340029bd0bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:29:31.624272 9309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c153e56492e41299f80bfa4116b124d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:31.624845 9155 tablet_service.cc:1467] Processing CreateTablet for tablet 3c153e56492e41299f80bfa4116b124d (DEFAULT_TABLE table=test-table [id=b18b713be14b4ea284406340029bd0bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:29:31.625635 9155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c153e56492e41299f80bfa4116b124d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:29:31.630481 9399 tablet_bootstrap.cc:492] T 3c153e56492e41299f80bfa4116b124d P 7fd4b95bd5fa48b3a647f21e63eeaee7: Bootstrap starting.
I20260808 18:29:31.631749 9401 tablet_bootstrap.cc:492] T 3c153e56492e41299f80bfa4116b124d P db4c6bed85f54abf9d81b0d8657a7831: Bootstrap starting.
I20260808 18:29:31.633198 9399 tablet_bootstrap.cc:654] T 3c153e56492e41299f80bfa4116b124d P 7fd4b95bd5fa48b3a647f21e63eeaee7: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:31.633947 9401 tablet_bootstrap.cc:654] T 3c153e56492e41299f80bfa4116b124d P db4c6bed85f54abf9d81b0d8657a7831: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:31.633929 9399 log.cc:824] T 3c153e56492e41299f80bfa4116b124d P 7fd4b95bd5fa48b3a647f21e63eeaee7: Log is configured to *not* fsync() on all Append() calls
I20260808 18:29:31.634729 9402 tablet_bootstrap.cc:492] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340: Bootstrap starting.
I20260808 18:29:31.634725 9401 log.cc:824] T 3c153e56492e41299f80bfa4116b124d P db4c6bed85f54abf9d81b0d8657a7831: Log is configured to *not* fsync() on all Append() calls
I20260808 18:29:31.635869 9399 tablet_bootstrap.cc:492] T 3c153e56492e41299f80bfa4116b124d P 7fd4b95bd5fa48b3a647f21e63eeaee7: No bootstrap required, opened a new log
I20260808 18:29:31.636080 9399 ts_tablet_manager.cc:1397] T 3c153e56492e41299f80bfa4116b124d P 7fd4b95bd5fa48b3a647f21e63eeaee7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:29:31.636641 9401 tablet_bootstrap.cc:492] T 3c153e56492e41299f80bfa4116b124d P db4c6bed85f54abf9d81b0d8657a7831: No bootstrap required, opened a new log
I20260808 18:29:31.636833 9401 ts_tablet_manager.cc:1397] T 3c153e56492e41299f80bfa4116b124d P db4c6bed85f54abf9d81b0d8657a7831: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 18:29:31.637142 9402 tablet_bootstrap.cc:654] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340: Neither blocks nor log segments found. Creating new log.
I20260808 18:29:31.637898 9402 log.cc:824] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340: Log is configured to *not* fsync() on all Append() calls
I20260808 18:29:31.639778 9402 tablet_bootstrap.cc:492] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340: No bootstrap required, opened a new log
I20260808 18:29:31.639966 9402 ts_tablet_manager.cc:1397] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 18:29:31.639954 9399 raft_consensus.cc:348] T 3c153e56492e41299f80bfa4116b124d P 7fd4b95bd5fa48b3a647f21e63eeaee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd4b95bd5fa48b3a647f21e63eeaee7" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39299 } } peers { permanent_uuid: "db4c6bed85f54abf9d81b0d8657a7831" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35775 } } peers { permanent_uuid: "8249e5b58bf44ed1ae5b85891a582340" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41509 } }
I20260808 18:29:31.640247 9399 raft_consensus.cc:374] T 3c153e56492e41299f80bfa4116b124d P 7fd4b95bd5fa48b3a647f21e63eeaee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:31.640311 9399 raft_consensus.cc:729] T 3c153e56492e41299f80bfa4116b124d P 7fd4b95bd5fa48b3a647f21e63eeaee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd4b95bd5fa48b3a647f21e63eeaee7, State: Initialized, Role: FOLLOWER
I20260808 18:29:31.640594 9401 raft_consensus.cc:348] T 3c153e56492e41299f80bfa4116b124d P db4c6bed85f54abf9d81b0d8657a7831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd4b95bd5fa48b3a647f21e63eeaee7" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39299 } } peers { permanent_uuid: "db4c6bed85f54abf9d81b0d8657a7831" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35775 } } peers { permanent_uuid: "8249e5b58bf44ed1ae5b85891a582340" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41509 } }
I20260808 18:29:31.640753 9399 consensus_queue.cc:260] T 3c153e56492e41299f80bfa4116b124d P 7fd4b95bd5fa48b3a647f21e63eeaee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd4b95bd5fa48b3a647f21e63eeaee7" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39299 } } peers { permanent_uuid: "db4c6bed85f54abf9d81b0d8657a7831" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35775 } } peers { permanent_uuid: "8249e5b58bf44ed1ae5b85891a582340" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41509 } }
I20260808 18:29:31.640839 9401 raft_consensus.cc:374] T 3c153e56492e41299f80bfa4116b124d P db4c6bed85f54abf9d81b0d8657a7831 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:31.640899 9401 raft_consensus.cc:729] T 3c153e56492e41299f80bfa4116b124d P db4c6bed85f54abf9d81b0d8657a7831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db4c6bed85f54abf9d81b0d8657a7831, State: Initialized, Role: FOLLOWER
I20260808 18:29:31.641410 9085 heartbeater.cc:503] Master 127.29.78.190:33977 was elected leader, sending a full tablet report...
I20260808 18:29:31.641733 9399 ts_tablet_manager.cc:1428] T 3c153e56492e41299f80bfa4116b124d P 7fd4b95bd5fa48b3a647f21e63eeaee7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 18:29:31.641932 9401 consensus_queue.cc:260] T 3c153e56492e41299f80bfa4116b124d P db4c6bed85f54abf9d81b0d8657a7831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd4b95bd5fa48b3a647f21e63eeaee7" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39299 } } peers { permanent_uuid: "db4c6bed85f54abf9d81b0d8657a7831" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35775 } } peers { permanent_uuid: "8249e5b58bf44ed1ae5b85891a582340" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41509 } }
I20260808 18:29:31.642724 9382 heartbeater.cc:503] Master 127.29.78.190:33977 was elected leader, sending a full tablet report...
I20260808 18:29:31.642944 9401 ts_tablet_manager.cc:1428] T 3c153e56492e41299f80bfa4116b124d P db4c6bed85f54abf9d81b0d8657a7831: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 18:29:31.643728 9402 raft_consensus.cc:348] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd4b95bd5fa48b3a647f21e63eeaee7" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39299 } } peers { permanent_uuid: "db4c6bed85f54abf9d81b0d8657a7831" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35775 } } peers { permanent_uuid: "8249e5b58bf44ed1ae5b85891a582340" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41509 } }
I20260808 18:29:31.644016 9402 raft_consensus.cc:374] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:29:31.644069 9402 raft_consensus.cc:729] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8249e5b58bf44ed1ae5b85891a582340, State: Initialized, Role: FOLLOWER
I20260808 18:29:31.644534 9402 consensus_queue.cc:260] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd4b95bd5fa48b3a647f21e63eeaee7" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39299 } } peers { permanent_uuid: "db4c6bed85f54abf9d81b0d8657a7831" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35775 } } peers { permanent_uuid: "8249e5b58bf44ed1ae5b85891a582340" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41509 } }
I20260808 18:29:31.645428 9402 ts_tablet_manager.cc:1428] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260808 18:29:31.645689 9233 heartbeater.cc:503] Master 127.29.78.190:33977 was elected leader, sending a full tablet report...
I20260808 18:29:31.807399 9409 raft_consensus.cc:482] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:29:31.807657 9409 raft_consensus.cc:504] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd4b95bd5fa48b3a647f21e63eeaee7" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39299 } } peers { permanent_uuid: "db4c6bed85f54abf9d81b0d8657a7831" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35775 } } peers { permanent_uuid: "8249e5b58bf44ed1ae5b85891a582340" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41509 } }
I20260808 18:29:31.808880 9409 leader_election.cc:288] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fd4b95bd5fa48b3a647f21e63eeaee7 (127.29.78.129:39299), db4c6bed85f54abf9d81b0d8657a7831 (127.29.78.131:35775)
W20260808 18:29:31.812587 9383 tablet.cc:2369] T 3c153e56492e41299f80bfa4116b124d P db4c6bed85f54abf9d81b0d8657a7831: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:29:31.813585 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:31.809322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:55915 (local address 127.29.78.129:39299)
0808 18:29:31.809480 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:31.809485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:31.809510 (+ 25us) server_negotiation.cc:407] Connection header received
0808 18:29:31.809558 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:31.809561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:31.809602 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:31.809670 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:31.810006 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.810816 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.812498 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.812993 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.813123 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:31.813176 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:31.813414 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 18:29:31.813486 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 18:29:31.813576 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:31.809082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:39299 (local address 127.29.78.130:55915)
0808 18:29:31.809227 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:31.809245 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:31.809364 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:31.809712 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:31.809715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:31.809750 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:31.809892 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:31.809898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.810946 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.810949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:31.812319 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:31.812358 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.813123 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.813127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:31.813243 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:31.813261 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 18:29:31.813351 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 18:29:31.813437 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":1,"spinlock_wait_cycles":5248}
I20260808 18:29:31.813680 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:31.809324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:58523 (local address 127.29.78.131:35775)
0808 18:29:31.809466 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:31.809470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:31.809732 (+ 262us) server_negotiation.cc:407] Connection header received
0808 18:29:31.809863 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:31.809866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:31.809908 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:31.810004 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:29:31.810239 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.810973 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.812465 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.813049 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.813168 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:31.813225 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:31.813478 (+ 253us) server_negotiation.cc:299] Negotiation successful
0808 18:29:31.813551 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 18:29:31.813738 9410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:31.809169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:35775 (local address 127.29.78.130:58523)
0808 18:29:31.809603 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:31.809614 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 18:29:31.809751 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:29:31.810030 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:29:31.810032 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:29:31.810049 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:29:31.810132 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:31.810135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.811078 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.811081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:31.812329 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:29:31.812338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.813237 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.813244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:29:31.813347 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:31.813364 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 18:29:31.813459 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 18:29:31.813548 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":55,"spinlock_wait_cycles":29568,"thread_start_us":141,"threads_started":1}
I20260808 18:29:31.814348 9332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c153e56492e41299f80bfa4116b124d" candidate_uuid: "8249e5b58bf44ed1ae5b85891a582340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db4c6bed85f54abf9d81b0d8657a7831" is_pre_election: true
I20260808 18:29:31.814359 9036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c153e56492e41299f80bfa4116b124d" candidate_uuid: "8249e5b58bf44ed1ae5b85891a582340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd4b95bd5fa48b3a647f21e63eeaee7" is_pre_election: true
I20260808 18:29:31.814692 9036 raft_consensus.cc:2454] T 3c153e56492e41299f80bfa4116b124d P 7fd4b95bd5fa48b3a647f21e63eeaee7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8249e5b58bf44ed1ae5b85891a582340 in term 0.
I20260808 18:29:31.814704 9332 raft_consensus.cc:2454] T 3c153e56492e41299f80bfa4116b124d P db4c6bed85f54abf9d81b0d8657a7831 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8249e5b58bf44ed1ae5b85891a582340 in term 0.
I20260808 18:29:31.815153 9109 leader_election.cc:302] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fd4b95bd5fa48b3a647f21e63eeaee7, 8249e5b58bf44ed1ae5b85891a582340; no voters:
I20260808 18:29:31.815448 9409 raft_consensus.cc:2781] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:29:31.815538 9409 raft_consensus.cc:482] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:29:31.815582 9409 raft_consensus.cc:3037] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:31.816638 9409 raft_consensus.cc:504] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd4b95bd5fa48b3a647f21e63eeaee7" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39299 } } peers { permanent_uuid: "db4c6bed85f54abf9d81b0d8657a7831" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35775 } } peers { permanent_uuid: "8249e5b58bf44ed1ae5b85891a582340" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41509 } }
I20260808 18:29:31.816987 9409 leader_election.cc:288] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340 [CANDIDATE]: Term 1 election: Requested vote from peers 7fd4b95bd5fa48b3a647f21e63eeaee7 (127.29.78.129:39299), db4c6bed85f54abf9d81b0d8657a7831 (127.29.78.131:35775)
I20260808 18:29:31.817312 9036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c153e56492e41299f80bfa4116b124d" candidate_uuid: "8249e5b58bf44ed1ae5b85891a582340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd4b95bd5fa48b3a647f21e63eeaee7"
I20260808 18:29:31.817395 9332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c153e56492e41299f80bfa4116b124d" candidate_uuid: "8249e5b58bf44ed1ae5b85891a582340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db4c6bed85f54abf9d81b0d8657a7831"
I20260808 18:29:31.817461 9036 raft_consensus.cc:3037] T 3c153e56492e41299f80bfa4116b124d P 7fd4b95bd5fa48b3a647f21e63eeaee7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:31.817536 9332 raft_consensus.cc:3037] T 3c153e56492e41299f80bfa4116b124d P db4c6bed85f54abf9d81b0d8657a7831 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:29:31.818569 9036 raft_consensus.cc:2454] T 3c153e56492e41299f80bfa4116b124d P 7fd4b95bd5fa48b3a647f21e63eeaee7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8249e5b58bf44ed1ae5b85891a582340 in term 1.
I20260808 18:29:31.818717 9332 raft_consensus.cc:2454] T 3c153e56492e41299f80bfa4116b124d P db4c6bed85f54abf9d81b0d8657a7831 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8249e5b58bf44ed1ae5b85891a582340 in term 1.
I20260808 18:29:31.818888 9109 leader_election.cc:302] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fd4b95bd5fa48b3a647f21e63eeaee7, 8249e5b58bf44ed1ae5b85891a582340; no voters:
I20260808 18:29:31.819067 9409 raft_consensus.cc:2781] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:29:31.819269 9409 raft_consensus.cc:686] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340 [term 1 LEADER]: Becoming Leader. State: Replica: 8249e5b58bf44ed1ae5b85891a582340, State: Running, Role: LEADER
I20260808 18:29:31.819664 9409 consensus_queue.cc:237] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd4b95bd5fa48b3a647f21e63eeaee7" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39299 } } peers { permanent_uuid: "db4c6bed85f54abf9d81b0d8657a7831" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35775 } } peers { permanent_uuid: "8249e5b58bf44ed1ae5b85891a582340" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41509 } }
I20260808 18:29:31.823014 8880 catalog_manager.cc:5495] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8249e5b58bf44ed1ae5b85891a582340 (127.29.78.130). New cstate: current_term: 1 leader_uuid: "8249e5b58bf44ed1ae5b85891a582340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fd4b95bd5fa48b3a647f21e63eeaee7" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db4c6bed85f54abf9d81b0d8657a7831" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8249e5b58bf44ed1ae5b85891a582340" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41509 } health_report { overall_health: HEALTHY } } }
W20260808 18:29:31.842741 9234 tablet.cc:2369] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 18:29:31.851274 9086 tablet.cc:2369] T 3c153e56492e41299f80bfa4116b124d P 7fd4b95bd5fa48b3a647f21e63eeaee7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:29:31.855943 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:31.852369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59230 (local address 127.29.78.130:41509)
0808 18:29:31.852525 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 18:29:31.852531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:29:31.852547 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:29:31.852635 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:29:31.852637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:29:31.852693 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:29:31.852807 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:29:31.853068 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.853574 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.854517 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:29:31.854732 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:29:31.854846 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:29:31.855160 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:29:31.855565 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:29:31.855705 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 18:29:31.855746 (+ 41us) server_negotiation.cc:299] Negotiation successful
0808 18:29:31.855801 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 18:29:31.861514 9332 raft_consensus.cc:1261] T 3c153e56492e41299f80bfa4116b124d P db4c6bed85f54abf9d81b0d8657a7831 [term 1 FOLLOWER]: Refusing update from remote peer 8249e5b58bf44ed1ae5b85891a582340: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:29:31.861514 9036 raft_consensus.cc:1261] T 3c153e56492e41299f80bfa4116b124d P 7fd4b95bd5fa48b3a647f21e63eeaee7 [term 1 FOLLOWER]: Refusing update from remote peer 8249e5b58bf44ed1ae5b85891a582340: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:29:31.862196 9409 consensus_queue.cc:1035] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db4c6bed85f54abf9d81b0d8657a7831" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:29:31.862418 9411 consensus_queue.cc:1035] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd4b95bd5fa48b3a647f21e63eeaee7" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:29:31.869669 9416 mvcc.cc:205] Tried to move back new op lower bound from 7316331609537851392 to 7316331609376370688. Current Snapshot: MvccSnapshot[applied={T|T < 7316331609537851392}]
I20260808 18:29:31.869884 9415 mvcc.cc:205] Tried to move back new op lower bound from 7316331609537851392 to 7316331609376370688. Current Snapshot: MvccSnapshot[applied={T|T < 7316331609537851392}]
I20260808 18:29:31.876245 8880 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:29:31.876382 8880 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:29:31.878592 8880 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8c4af6719dc4dedbcf671cd2f8581ff: Sending DeleteTablet for 3 replicas of tablet 3c153e56492e41299f80bfa4116b124d
I20260808 18:29:31.879292 9015 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c153e56492e41299f80bfa4116b124d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:29:31 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:36396
I20260808 18:29:31.879361 9155 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c153e56492e41299f80bfa4116b124d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:29:31 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:59226
I20260808 18:29:31.879774 9309 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c153e56492e41299f80bfa4116b124d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:29:31 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:57480
I20260808 18:29:31.880081 9425 tablet_replica.cc:330] T 3c153e56492e41299f80bfa4116b124d P 7fd4b95bd5fa48b3a647f21e63eeaee7: stopping tablet replica
I20260808 18:29:31.880214 9426 tablet_replica.cc:330] T 3c153e56492e41299f80bfa4116b124d P db4c6bed85f54abf9d81b0d8657a7831: stopping tablet replica
I20260808 18:29:31.880251 9423 tablet_replica.cc:330] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340: stopping tablet replica
I20260808 18:29:31.880538 9426 raft_consensus.cc:2229] T 3c153e56492e41299f80bfa4116b124d P db4c6bed85f54abf9d81b0d8657a7831 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:29:31.880570 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 8948
I20260808 18:29:31.880837 9426 raft_consensus.cc:2258] T 3c153e56492e41299f80bfa4116b124d P db4c6bed85f54abf9d81b0d8657a7831 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:29:31.881004 9423 raft_consensus.cc:2229] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:29:31.881414 9423 raft_consensus.cc:2258] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:29:31.883037 9426 ts_tablet_manager.cc:1905] T 3c153e56492e41299f80bfa4116b124d P db4c6bed85f54abf9d81b0d8657a7831: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:29:31.883657 9423 ts_tablet_manager.cc:1905] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:29:31.886307 9423 ts_tablet_manager.cc:1918] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:29:31.886485 9423 log.cc:1196] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/3c153e56492e41299f80bfa4116b124d
I20260808 18:29:31.886885 9423 ts_tablet_manager.cc:1939] T 3c153e56492e41299f80bfa4116b124d P 8249e5b58bf44ed1ae5b85891a582340: Deleting consensus metadata
I20260808 18:29:31.886936 9426 ts_tablet_manager.cc:1918] T 3c153e56492e41299f80bfa4116b124d P db4c6bed85f54abf9d81b0d8657a7831: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:29:31.887068 9426 log.cc:1196] T 3c153e56492e41299f80bfa4116b124d P db4c6bed85f54abf9d81b0d8657a7831: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/3c153e56492e41299f80bfa4116b124d
I20260808 18:29:31.887502 9426 ts_tablet_manager.cc:1939] T 3c153e56492e41299f80bfa4116b124d P db4c6bed85f54abf9d81b0d8657a7831: Deleting consensus metadata
I20260808 18:29:31.888255 8867 catalog_manager.cc:4841] TS 8249e5b58bf44ed1ae5b85891a582340 (127.29.78.130:41509): tablet 3c153e56492e41299f80bfa4116b124d (table test-table [id=b18b713be14b4ea284406340029bd0bb]) successfully deleted
I20260808 18:29:31.888913 8868 catalog_manager.cc:4841] TS db4c6bed85f54abf9d81b0d8657a7831 (127.29.78.131:35775): tablet 3c153e56492e41299f80bfa4116b124d (table test-table [id=b18b713be14b4ea284406340029bd0bb]) successfully deleted
W20260808 18:29:31.895071 8869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:39299 (error 108)
I20260808 18:29:31.895993 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 9090
I20260808 18:29:31.896186 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:29:31.895854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:39299 (local address 127.0.0.1:36410)
0808 18:29:31.896000 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 18:29:31.896085 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:39299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 18:29:31.897853 8869 catalog_manager.cc:4584] TS 7fd4b95bd5fa48b3a647f21e63eeaee7 (127.29.78.129:39299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c153e56492e41299f80bfa4116b124d: Network error: Client connection negotiation failed: client connection to 127.29.78.129:39299: connect: Connection refused (error 111)
I20260808 18:29:31.903281 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 9238
I20260808 18:29:31.910570 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 8849
2026-08-08T18:29:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2588 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:29:31 dist-test-slave-gp6n krb5kdc[9430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:29:31 dist-test-slave-gp6n krb5kdc[9430](info): set up 2 sockets
Aug 08 18:29:31 dist-test-slave-gp6n krb5kdc[9430](info): commencing operation
krb5kdc: starting...
W20260808 18:29:33.019114 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:29:33 dist-test-slave-gp6n krb5kdc[9430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:29:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:29:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.29.78.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.29.78.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/rangeradmin_127.29.78.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/rangeradmin_127.29.78.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.29.78.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.29.78.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/rangerlookup_127.29.78.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/rangerlookup_127.29.78.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.78.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.29.78.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/HTTP_127.29.78.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/HTTP_127.29.78.148@KRBTEST.COM.keytab.
I20260808 18:29:33.197715 30010 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task0kHl76/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/postgres -l logfile start
2026-08-08 18:29:35.830 UTC [9463] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 18:29:35.830 UTC [9463] LOG: listening on IPv4 address "127.29.78.148", port 49213
2026-08-08 18:29:35.833 UTC [9463] LOG: listening on Unix socket "/tmp/.s.PGSQL.49213"
2026-08-08 18:29:35.869 UTC [9467] LOG: database system was shut down at 2026-08-08 18:29:34 UTC
2026-08-08 18:29:35.874 UTC [9463] LOG: database system is ready to accept connections
I20260808 18:29:36.870500 30010 mini_postgres.cc:96] Postgres bound to 49213
2026-08-08 18:29:36.878 UTC [9476] FATAL: database "slave" does not exist
127.29.78.148:49213 - accepting connections
I20260808 18:29:36.879501 30010 mini_ranger.cc:161] Starting Ranger...
I20260808 18:29:36.895166 30010 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 18:29:37.403486 30010 mini_ranger.cc:87] Created ranger Postgres database
I20260808 18:29:37.403606 30010 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ranger-admin
2026-08-08 18:29:37,732 [I] DB FLAVOR :POSTGRES
2026-08-08 18:29:37,732 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 18:29:37,732 [I] Checking connection..
2026-08-08 18:29:37,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 18:29:38,000 [I] Checking connection passed.
2026-08-08 18:29:38,001 [I] --------- Verifying version history table ---------
2026-08-08 18:29:38,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 18:29:38,268 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 18:29:38,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 18:29:38,531 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 18:29:38,531 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 18:29:38,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 18:29:38.788 UTC [9547] WARNING: there is no transaction in progress
2026-08-08 18:29:38,795 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 18:29:38,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 18:29:39,057 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 18:29:39,057 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 18:29:39,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 18:29:39,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 18:29:39,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gp6n.c.gcp-upstream.internal','N') ;"
2026-08-08 18:29:39,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 18:29:40,042 [I] Table x_portal_user does not exist in database ranger
2026-08-08 18:29:40,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 18:29:40,298 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 18:29:40,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 18:29:40,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 18:29:40,781 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 18:29:40,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 18:29:41.743 UTC [9668] WARNING: there is no transaction in progress
2026-08-08 18:29:41.758 UTC [9668] WARNING: there is no transaction in progress
2026-08-08 18:29:41.775 UTC [9668] WARNING: there is no transaction in progress
2026-08-08 18:29:41.791 UTC [9668] WARNING: there is no transaction in progress
2026-08-08 18:29:41.806 UTC [9668] WARNING: there is no transaction in progress
2026-08-08 18:29:41.824 UTC [9668] WARNING: there is no transaction in progress
2026-08-08 18:29:41.880 UTC [9668] WARNING: there is no transaction in progress
2026-08-08 18:29:41.886 UTC [9668] WARNING: there is no transaction in progress
2026-08-08 18:29:41.900 UTC [9668] WARNING: there is no transaction in progress
2026-08-08 18:29:41.910 UTC [9668] WARNING: there is no transaction in progress
2026-08-08 18:29:41.920 UTC [9668] WARNING: there is no transaction in progress
2026-08-08 18:29:41.933 UTC [9668] WARNING: there is no transaction in progress
2026-08-08 18:29:41.943 UTC [9668] WARNING: there is no transaction in progress
2026-08-08 18:29:41.949 UTC [9668] WARNING: there is no transaction in progress
2026-08-08 18:29:41.957 UTC [9668] WARNING: there is no transaction in progress
2026-08-08 18:29:41.964 UTC [9668] WARNING: there is no transaction in progress
2026-08-08 18:29:42,623 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 18:29:42,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 18:29:42,873 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 18:29:42,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 18:29:43,129 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 18:29:43,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 18:29:43,367 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 18:29:43,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 18:29:43,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 18:29:43,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gp6n.c.gcp-upstream.internal';"
2026-08-08 18:29:44,091 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 18:29:44,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 18:29:44,339 [I] DB_PATCHES have already been applied
I20260808 18:29:44.344776 30010 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ranger-admin:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task0kHl76/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 18:29:44.344846 30010 mini_ranger.cc:193] Using host: 127.29.78.148
I20260808 18:29:44.349224 30010 mini_ranger.cc:227] Ranger admin URL: http://127.29.78.148:38909
Aug 08, 2026 6:29:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 6:29:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 6:29:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 6:29:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 6:29:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 6:29:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 6:29:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 18:29:45.534118 30010 mini_ranger.cc:160] Time spent starting Ranger: real 8.655s user 0.006s sys 0.006s
Aug 08, 2026 6:30:05 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 6:30:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 6:30:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 6:30:05 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 6:30:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 6:30:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 6:30:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 6:30:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 6:30:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 6:30:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 6:30:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 6:30:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 6:30:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 6:30:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 6:30:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 6:30:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 6:30:05 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 6:30:06 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 18:30:07.389671 30010 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.29.78.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.29.78.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/rangerkms_127.29.78.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/rangerkms_127.29.78.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.78.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.29.78.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.78.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/HTTP_127.29.78.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.78.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/HTTP_127.29.78.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 18:30:07 dist-test-slave-gp6n krb5kdc[9430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213807, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 18:30:07.483268 30010 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 18:30:07.493482 30010 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 18:30:08.359344 30010 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 18:30:08.359450 30010 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ranger-kms
I20260808 18:30:08.359565 30010 mini_ranger_kms.cc:227] Using postgres at 127.29.78.148:49213
2026-08-08 18:30:08,600 [I] DB FLAVOR :POSTGRES
2026-08-08 18:30:08,600 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 18:30:08,600 [I] Checking connection
2026-08-08 18:30:08,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 18:30:08,835 [I] connection success
2026-08-08 18:30:08,835 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 18:30:08,836 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 18:30:08,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 18:30:09,096 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 18:30:09,096 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 18:30:09,096 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 18:30:09,096 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:49213/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 18:30:09,354 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 18:30:09.795828 30010 mini_ranger_kms.cc:310] Created kms service
I20260808 18:30:09.971926 30010 mini_ranger_kms.cc:326] Created kudu user
I20260808 18:30:10.017907 30010 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 18:30:10 dist-test-slave-gp6n krb5kdc[9430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213807, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260808 18:30:10.457268 30010 mini_ranger_kms.cc:384] Added ranger policy
I20260808 18:30:10.457438 30010 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ranger-kms:/tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 18:30:10.457463 30010 mini_ranger_kms.cc:240] Using host: 127.29.78.148
I20260808 18:30:10.462785 30010 mini_ranger_kms.cc:276] Ranger KMS PID: 9861
I20260808 18:30:10.462918 30010 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.29.78.148:41547
Aug 08, 2026 6:30:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 6:30:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 6:30:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 6:30:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 6:30:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 6:30:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260808 18:30:12.866122 30010 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.383s user 0.019s sys 0.206s
I20260808 18:30:12.866283 30010 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 18:30:12.866449 30010 mini_ranger_kms.cc:401] 127.29.78.148:41547/kms/v1/keys
Aug 08 18:30:12 dist-test-slave-gp6n krb5kdc[9430](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786213812, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.78.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 08 18:30:13 dist-test-slave-gp6n krb5kdc[9430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213812, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.78.148@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
Aug 08 18:30:13 dist-test-slave-gp6n krb5kdc[9430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213812, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.78.148@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
Aug 08 18:30:16 dist-test-slave-gp6n krb5kdc[9430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:30:16.050262 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:42713
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:44179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:42713
--ranger_config_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:41547/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:30:16.164352 9920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:30:16.164639 9920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:30:16.164680 9920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:30:16.168836 9920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:30:16.168905 9920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:30:16.168926 9920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:30:16.168944 9920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:30:16.168959 9920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:30:16.174706 9920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/master-0/wal
--ranger_kms_url=127.29.78.148:41547/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:42713
--ranger_config_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:42713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:30:16.175846 9920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:30:16.176994 9920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:30:16.186086 9925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:30:16.186072 9928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:30:16.186285 9920 server_base.cc:1034] running on GCE node
W20260808 18:30:16.186131 9926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:30:16.187013 9920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:30:16.188149 9920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:30:16.189343 9920 hybrid_clock.cc:648] HybridClock initialized: now 1786213816189318 us; error 45 us; skew 500 ppm
Aug 08 18:30:16 dist-test-slave-gp6n krb5kdc[9430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213816, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:30:16.198552 9920 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:30:16.199525 9920 webserver.cc:460] Webserver started at http://127.29.78.190:43175/ using document root <none> and password file <none>
I20260808 18:30:16.200094 9920 fs_manager.cc:359] Metadata directory not provided
I20260808 18:30:16.200178 9920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:30:16.200389 9920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 18:30:16 dist-test-slave-gp6n krb5kdc[9430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213816, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260808 18:30:16.254204 9920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "b59d9fde4d574db0970ac3fdc051127f"
format_stamp: "Formatted at 2026-08-08 18:30:16 on dist-test-slave-gp6n"
server_key: "898a8e18f753103cbf98d8d845307762"
server_key_iv: "be9ff76749529b5135538f177b3083eb"
server_key_version: "kuduclusterkey@0"
I20260808 18:30:16.254999 9920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "b59d9fde4d574db0970ac3fdc051127f"
format_stamp: "Formatted at 2026-08-08 18:30:16 on dist-test-slave-gp6n"
server_key: "898a8e18f753103cbf98d8d845307762"
server_key_iv: "be9ff76749529b5135538f177b3083eb"
server_key_version: "kuduclusterkey@0"
I20260808 18:30:16.259347 9920 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 18:30:16.274978 9937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:30:16.276595 9920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:30:16.276733 9920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "b59d9fde4d574db0970ac3fdc051127f"
format_stamp: "Formatted at 2026-08-08 18:30:16 on dist-test-slave-gp6n"
server_key: "898a8e18f753103cbf98d8d845307762"
server_key_iv: "be9ff76749529b5135538f177b3083eb"
server_key_version: "kuduclusterkey@0"
I20260808 18:30:16.276858 9920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:30:16.294015 9920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:30:16.297223 9920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:30:16.297434 9920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:30:16.305996 9920 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:42713
I20260808 18:30:16.306062 9992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:42713 every 8 connection(s)
I20260808 18:30:16.307294 9920 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:30:16.310096 9993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:30:16.312199 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 9920
I20260808 18:30:16.312336 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:30:16.312649 30010 external_mini_cluster.cc:1428] Setting key a3a0a432dd793a1695b2f2f26f1a5d48
I20260808 18:30:16.317317 9993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f: Bootstrap starting.
I20260808 18:30:16.319872 9993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f: Neither blocks nor log segments found. Creating new log.
I20260808 18:30:16.320776 9993 log.cc:824] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f: Log is configured to *not* fsync() on all Append() calls
I20260808 18:30:16.323061 9993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f: No bootstrap required, opened a new log
Aug 08 18:30:16 dist-test-slave-gp6n krb5kdc[9430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:30:16.326555 9993 raft_consensus.cc:348] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59d9fde4d574db0970ac3fdc051127f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42713 } }
I20260808 18:30:16.326820 9993 raft_consensus.cc:374] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:30:16.326937 9993 raft_consensus.cc:729] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b59d9fde4d574db0970ac3fdc051127f, State: Initialized, Role: FOLLOWER
I20260808 18:30:16.327524 9993 consensus_queue.cc:260] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59d9fde4d574db0970ac3fdc051127f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42713 } }
I20260808 18:30:16.327729 9993 raft_consensus.cc:388] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:30:16.327857 9993 raft_consensus.cc:482] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:30:16.327970 9993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:30:16.329169 9993 raft_consensus.cc:504] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59d9fde4d574db0970ac3fdc051127f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42713 } }
I20260808 18:30:16.329535 9993 leader_election.cc:302] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b59d9fde4d574db0970ac3fdc051127f; no voters:
I20260808 18:30:16.329873 9993 leader_election.cc:288] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:30:16.329999 9999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:30:16.330171 9997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:30:16.314149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55544 (local address 127.29.78.190:42713)
0808 18:30:16.314620 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0808 18:30:16.314642 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 18:30:16.314686 (+ 44us) server_negotiation.cc:407] Connection header received
0808 18:30:16.315483 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:30:16.315506 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:30:16.315930 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:30:16.316315 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:30:16.317210 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:30:16.318658 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:30:16.319904 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:30:16.320596 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:30:16.320755 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:30:16.323889 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:30:16.323897 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:30:16.323913 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:30:16.323943 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:30:16.326651 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:30:16.327212 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:30:16.327218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:30:16.327225 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:30:16.327358 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:30:16.327704 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:30:16.327707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:30:16.327709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:30:16.328363 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:30:16.328558 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0808 18:30:16.328871 (+ 313us) server_negotiation.cc:299] Negotiation successful
0808 18:30:16.329170 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":158,"threads_started":1}
I20260808 18:30:16.330274 9999 raft_consensus.cc:686] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f [term 1 LEADER]: Becoming Leader. State: Replica: b59d9fde4d574db0970ac3fdc051127f, State: Running, Role: LEADER
I20260808 18:30:16.330772 9999 consensus_queue.cc:237] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59d9fde4d574db0970ac3fdc051127f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42713 } }
I20260808 18:30:16.331282 9993 sys_catalog.cc:564] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:30:16.331949 9999 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b59d9fde4d574db0970ac3fdc051127f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59d9fde4d574db0970ac3fdc051127f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42713 } } }
I20260808 18:30:16.332057 9999 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f [sys.catalog]: This master's current role is: LEADER
I20260808 18:30:16.333305 10001 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b59d9fde4d574db0970ac3fdc051127f. Latest consensus state: current_term: 1 leader_uuid: "b59d9fde4d574db0970ac3fdc051127f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59d9fde4d574db0970ac3fdc051127f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42713 } } }
I20260808 18:30:16.333482 10001 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f [sys.catalog]: This master's current role is: LEADER
I20260808 18:30:16.333781 10003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:30:16.336951 10003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:30:16.337628 9993 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 18:30:16.347407 10003 catalog_manager.cc:1347] Generated new cluster ID: df0aecf90e7944ee84f794d253c04749
I20260808 18:30:16.347487 10003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:30:16.366248 10003 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:30:16.368233 10003 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:30:16.380479 10003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f: Generated new TSK 0
I20260808 18:30:16.381467 10003 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260808 18:30:17.170971 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:41547/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:42713
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:44179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:30:17.318369 10044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:30:17.318800 10044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:30:17.318915 10044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:30:17.323263 10044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:30:17.323341 10044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:30:17.323427 10044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:30:17.328961 10044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.29.78.148:41547/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:42713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:30:17.330191 10044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:30:17.331470 10044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:30:17.342864 10054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:30:17.342917 10052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:30:17.343264 10051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:30:17.343532 10044 server_base.cc:1034] running on GCE node
I20260808 18:30:17.343986 10044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:30:17.344785 10044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:30:17.345990 10044 hybrid_clock.cc:648] HybridClock initialized: now 1786213817345969 us; error 34 us; skew 500 ppm
Aug 08 18:30:17 dist-test-slave-gp6n krb5kdc[9430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213817, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:30:17.349792 10044 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:30:17.350946 10044 webserver.cc:460] Webserver started at http://127.29.78.129:44435/ using document root <none> and password file <none>
I20260808 18:30:17.351658 10044 fs_manager.cc:359] Metadata directory not provided
I20260808 18:30:17.351751 10044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:30:17.351996 10044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 18:30:17 dist-test-slave-gp6n krb5kdc[9430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213817, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260808 18:30:17.368070 10044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "858a38791b62436dbccf8eb47d4848dd"
format_stamp: "Formatted at 2026-08-08 18:30:17 on dist-test-slave-gp6n"
server_key: "0d5cda861d1a7da3e7a3a4479499e0bd"
server_key_iv: "d0bcf9f5c1dc07e047abf184b341ac78"
server_key_version: "kuduclusterkey@0"
I20260808 18:30:17.368852 10044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "858a38791b62436dbccf8eb47d4848dd"
format_stamp: "Formatted at 2026-08-08 18:30:17 on dist-test-slave-gp6n"
server_key: "0d5cda861d1a7da3e7a3a4479499e0bd"
server_key_iv: "d0bcf9f5c1dc07e047abf184b341ac78"
server_key_version: "kuduclusterkey@0"
I20260808 18:30:17.374961 10044 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 18:30:17.404009 10062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:30:17.406395 10044 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.000s
I20260808 18:30:17.406592 10044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "858a38791b62436dbccf8eb47d4848dd"
format_stamp: "Formatted at 2026-08-08 18:30:17 on dist-test-slave-gp6n"
server_key: "0d5cda861d1a7da3e7a3a4479499e0bd"
server_key_iv: "d0bcf9f5c1dc07e047abf184b341ac78"
server_key_version: "kuduclusterkey@0"
I20260808 18:30:17.406764 10044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:30:17.432853 10044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:30:17.437074 10044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:30:17.437366 10044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:30:17.438282 10044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:30:17.439661 10044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:30:17.439745 10044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:30:17.439805 10044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:30:17.439885 10044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:30:17.454109 10044 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:46255
I20260808 18:30:17.455528 10044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:30:17.463855 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 10044
I20260808 18:30:17.464012 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:30:17.464378 30010 external_mini_cluster.cc:1428] Setting key 2776f0ac37305789cd898e6dbeb3ca97
I20260808 18:30:17.463536 10189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:46255 every 8 connection(s)
Aug 08 18:30:17 dist-test-slave-gp6n krb5kdc[9430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213817, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:30:17.478025 10193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:30:17.460043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:43069 (local address 127.29.78.190:42713)
0808 18:30:17.460573 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0808 18:30:17.460578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:30:17.461933 (+ 1355us) server_negotiation.cc:407] Connection header received
0808 18:30:17.462325 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:30:17.462331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:30:17.462443 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:30:17.462551 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:30:17.464495 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:30:17.466316 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:30:17.468560 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:30:17.468970 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:30:17.469081 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:30:17.472156 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:30:17.472160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:30:17.472166 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:30:17.472195 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:30:17.474030 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:30:17.474858 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:30:17.474865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:30:17.474871 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:30:17.474975 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:30:17.475510 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:30:17.475517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:30:17.475522 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:30:17.475800 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:30:17.475925 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 18:30:17.477704 (+ 1779us) server_negotiation.cc:299] Negotiation successful
0808 18:30:17.477844 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":154,"threads_started":1}
I20260808 18:30:17.478464 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:30:17.458842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42713 (local address 127.29.78.129:43069)
0808 18:30:17.460602 (+ 1760us) negotiation.cc:107] Waiting for socket to connect
0808 18:30:17.460664 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 18:30:17.461949 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:30:17.462903 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:30:17.462912 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:30:17.463314 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:30:17.464214 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:30:17.464241 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:30:17.466528 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:30:17.466535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:30:17.468362 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:30:17.468387 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:30:17.469264 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:30:17.469269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:30:17.469444 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:30:17.470206 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:30:17.470224 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:30:17.471978 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:30:17.474176 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:30:17.474182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:30:17.474198 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:30:17.474721 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:30:17.475092 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:30:17.475098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:30:17.475104 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:30:17.475308 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:30:17.475990 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:30:17.476000 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:30:17.476409 (+ 409us) client_negotiation.cc:769] Sending connection context
0808 18:30:17.476664 (+ 255us) client_negotiation.cc:240] Negotiation successful
0808 18:30:17.477076 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1497,"thread_start_us":148,"threads_started":1}
I20260808 18:30:17.479817 10190 heartbeater.cc:347] Connected to a master server at 127.29.78.190:42713
I20260808 18:30:17.480079 10190 heartbeater.cc:464] Registering TS with master...
I20260808 18:30:17.480639 10190 heartbeater.cc:511] Master 127.29.78.190:42713 requested a full tablet report, sending...
I20260808 18:30:17.482370 9955 ts_manager.cc:194] Registered new tserver with Master: 858a38791b62436dbccf8eb47d4848dd (127.29.78.129:46255)
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260808 18:30:17.484213 9955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:43069
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260808 18:30:17.509624 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:41547/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:42713
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:44179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 18:30:17 dist-test-slave-gp6n krb5kdc[9430](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786213817, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 18:30:17.645758 10196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:30:17.646049 10196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:30:17.646183 10196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:30:17.650347 10196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:30:17.650476 10196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:30:17.650602 10196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:30:17.656098 10196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.29.78.148:41547/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:42713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:30:17.657323 10196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:30:17.658380 10196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:30:17.668344 10206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:30:17.668413 10203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:30:17.668759 10204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:30:17.669088 10196 server_base.cc:1034] running on GCE node
I20260808 18:30:17.669493 10196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:30:17.670143 10196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:30:17.671312 10196 hybrid_clock.cc:648] HybridClock initialized: now 1786213817671285 us; error 44 us; skew 500 ppm
Aug 08 18:30:17 dist-test-slave-gp6n krb5kdc[9430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213817, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:30:17.674238 10196 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:30:17.675237 10196 webserver.cc:460] Webserver started at http://127.29.78.130:41737/ using document root <none> and password file <none>
I20260808 18:30:17.675969 10196 fs_manager.cc:359] Metadata directory not provided
I20260808 18:30:17.676092 10196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:30:17.676348 10196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 18:30:17 dist-test-slave-gp6n krb5kdc[9430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213817, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260808 18:30:17.693409 10196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "4210311bae3742e1887b08b1b940aefe"
format_stamp: "Formatted at 2026-08-08 18:30:17 on dist-test-slave-gp6n"
server_key: "38be7af36e5b860e8d9ad65765b3f6aa"
server_key_iv: "6d3037acf27785c74f6a1f73f951d6ac"
server_key_version: "kuduclusterkey@0"
I20260808 18:30:17.694095 10196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "4210311bae3742e1887b08b1b940aefe"
format_stamp: "Formatted at 2026-08-08 18:30:17 on dist-test-slave-gp6n"
server_key: "38be7af36e5b860e8d9ad65765b3f6aa"
server_key_iv: "6d3037acf27785c74f6a1f73f951d6ac"
server_key_version: "kuduclusterkey@0"
I20260808 18:30:17.699855 10196 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260808 18:30:17.718937 10213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:30:17.720603 10196 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260808 18:30:17.720746 10196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "4210311bae3742e1887b08b1b940aefe"
format_stamp: "Formatted at 2026-08-08 18:30:17 on dist-test-slave-gp6n"
server_key: "38be7af36e5b860e8d9ad65765b3f6aa"
server_key_iv: "6d3037acf27785c74f6a1f73f951d6ac"
server_key_version: "kuduclusterkey@0"
I20260808 18:30:17.720880 10196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:30:17.741353 10196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:30:17.745170 10196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:30:17.745416 10196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:30:17.746162 10196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:30:17.747264 10196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:30:17.747376 10196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:30:17.747481 10196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:30:17.747551 10196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:30:17.768523 10196 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:39375
I20260808 18:30:17.769801 10196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:30:17.773455 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 10196
I20260808 18:30:17.773581 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:30:17.773893 30010 external_mini_cluster.cc:1428] Setting key 129450d94471ac24a7b0fc7d4f99dc80
I20260808 18:30:17.779314 10339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:39375 every 8 connection(s)
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
Aug 08 18:30:17 dist-test-slave-gp6n krb5kdc[9430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213817, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:30:17.804690 10193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:30:17.783164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:58139 (local address 127.29.78.190:42713)
0808 18:30:17.783285 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 18:30:17.783290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:30:17.784624 (+ 1334us) server_negotiation.cc:407] Connection header received
0808 18:30:17.785896 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:30:17.785900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:30:17.785963 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:30:17.786019 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:30:17.787708 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:30:17.788556 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:30:17.790280 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:30:17.790645 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:30:17.790768 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:30:17.800075 (+ 9307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:30:17.800080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:30:17.800083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:30:17.800115 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:30:17.802479 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:30:17.803021 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:30:17.803025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:30:17.803027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:30:17.803111 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:30:17.803535 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:30:17.803537 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:30:17.803538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:30:17.803727 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:30:17.803829 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 18:30:17.804367 (+ 538us) server_negotiation.cc:299] Negotiation successful
0808 18:30:17.804515 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 18:30:17.805728 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:30:17.781884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42713 (local address 127.29.78.130:58139)
0808 18:30:17.784466 (+ 2582us) negotiation.cc:107] Waiting for socket to connect
0808 18:30:17.784526 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 18:30:17.785619 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:30:17.786468 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:30:17.786478 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:30:17.786840 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:30:17.787536 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:30:17.787554 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:30:17.788715 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:30:17.788719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:30:17.790120 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:30:17.790132 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:30:17.790965 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:30:17.790970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:30:17.791180 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:30:17.797976 (+ 6796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:30:17.798001 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:30:17.799828 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:30:17.802605 (+ 2777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:30:17.802616 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:30:17.802630 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:30:17.802897 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:30:17.803203 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:30:17.803207 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:30:17.803209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:30:17.803368 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:30:17.803843 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:30:17.803852 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:30:17.804120 (+ 268us) client_negotiation.cc:769] Sending connection context
0808 18:30:17.804309 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 18:30:17.804649 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2131,"thread_start_us":275,"threads_started":1}
I20260808 18:30:17.807104 10340 heartbeater.cc:347] Connected to a master server at 127.29.78.190:42713
I20260808 18:30:17.807374 10340 heartbeater.cc:464] Registering TS with master...
I20260808 18:30:17.808017 10340 heartbeater.cc:511] Master 127.29.78.190:42713 requested a full tablet report, sending...
I20260808 18:30:17.809047 9955 ts_manager.cc:194] Registered new tserver with Master: 4210311bae3742e1887b08b1b940aefe (127.29.78.130:39375)
I20260808 18:30:17.809634 9955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:58139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260808 18:30:17.829507 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:41547/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:42713
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:44179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 18:30:17 dist-test-slave-gp6n krb5kdc[9430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213817, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
W20260808 18:30:17.996410 10345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:30:17.996676 10345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:30:17.996764 10345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:30:18.001673 10345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:30:18.001801 10345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:30:18.001928 10345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:30:18.009282 10345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.29.78.148:41547/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:42713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:30:18.010886 10345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:30:18.012718 10345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:30:18.026080 10345 server_base.cc:1034] running on GCE node
W20260808 18:30:18.027563 10350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:30:18.028208 10351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:30:18.028458 10355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:30:18.029145 10345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:30:18.029963 10345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:30:18.031203 10345 hybrid_clock.cc:648] HybridClock initialized: now 1786213818031115 us; error 100 us; skew 500 ppm
Aug 08 18:30:18 dist-test-slave-gp6n krb5kdc[9430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213818, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:30:18.034933 10345 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:30:18.036057 10345 webserver.cc:460] Webserver started at http://127.29.78.131:33589/ using document root <none> and password file <none>
I20260808 18:30:18.036736 10345 fs_manager.cc:359] Metadata directory not provided
I20260808 18:30:18.036824 10345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:30:18.037094 10345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 18:30:18 dist-test-slave-gp6n krb5kdc[9430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213818, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260808 18:30:18.057840 10345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "027224b7aa174cc4a7f0dc4ee97b7645"
format_stamp: "Formatted at 2026-08-08 18:30:18 on dist-test-slave-gp6n"
server_key: "096ca14afa6160d54da4d1225117716e"
server_key_iv: "d9d30f0c775810e2e0046b93a7a5d80e"
server_key_version: "kuduclusterkey@0"
I20260808 18:30:18.058957 10345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "027224b7aa174cc4a7f0dc4ee97b7645"
format_stamp: "Formatted at 2026-08-08 18:30:18 on dist-test-slave-gp6n"
server_key: "096ca14afa6160d54da4d1225117716e"
server_key_iv: "d9d30f0c775810e2e0046b93a7a5d80e"
server_key_version: "kuduclusterkey@0"
I20260808 18:30:18.065222 10345 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 18:30:18.085402 10364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:30:18.087198 10345 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260808 18:30:18.087328 10345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "027224b7aa174cc4a7f0dc4ee97b7645"
format_stamp: "Formatted at 2026-08-08 18:30:18 on dist-test-slave-gp6n"
server_key: "096ca14afa6160d54da4d1225117716e"
server_key_iv: "d9d30f0c775810e2e0046b93a7a5d80e"
server_key_version: "kuduclusterkey@0"
I20260808 18:30:18.087473 10345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:30:18.132586 10345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:30:18.136448 10345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:30:18.136699 10345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:30:18.137401 10345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:30:18.138552 10345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:30:18.138672 10345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:30:18.138769 10345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:30:18.138832 10345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:30:18.155660 10345 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:38015
I20260808 18:30:18.155707 10486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:38015 every 8 connection(s)
I20260808 18:30:18.156931 10345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:30:18.166534 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 10345
I20260808 18:30:18.166673 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:30:18.167013 30010 external_mini_cluster.cc:1428] Setting key 23468b60d04b4aff678efb087b3d5b44
Aug 08 18:30:18 dist-test-slave-gp6n krb5kdc[9430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213818, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:30:18.174088 10193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:30:18.159161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:38403 (local address 127.29.78.190:42713)
0808 18:30:18.159311 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 18:30:18.159316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:30:18.161183 (+ 1867us) server_negotiation.cc:407] Connection header received
0808 18:30:18.162332 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:30:18.162336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:30:18.162394 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:30:18.162528 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:30:18.163905 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.164738 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.166432 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.166780 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.166857 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:30:18.169874 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:30:18.169878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:30:18.169882 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:30:18.169912 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:30:18.171751 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:30:18.172332 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:30:18.172335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:30:18.172337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:30:18.172432 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:30:18.172820 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:30:18.172823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:30:18.172825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:30:18.173068 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:30:18.173169 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 18:30:18.173765 (+ 596us) server_negotiation.cc:299] Negotiation successful
0808 18:30:18.173913 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 18:30:18.175034 10492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:30:18.159634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42713 (local address 127.29.78.131:38403)
0808 18:30:18.160897 (+ 1263us) negotiation.cc:107] Waiting for socket to connect
0808 18:30:18.161044 (+ 147us) client_negotiation.cc:174] Beginning negotiation
0808 18:30:18.162088 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:30:18.162693 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:30:18.162702 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:30:18.163053 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:30:18.163740 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:30:18.163759 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.164872 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.164876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:30:18.166253 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:30:18.166263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.167054 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.167057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:30:18.167259 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:30:18.167985 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:30:18.168005 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:30:18.169692 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:30:18.171873 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:30:18.171879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:30:18.171894 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:30:18.172225 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:30:18.172539 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:30:18.172541 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:30:18.172544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:30:18.172720 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:30:18.173188 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:30:18.173194 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:30:18.173497 (+ 303us) client_negotiation.cc:769] Sending connection context
0808 18:30:18.173695 (+ 198us) client_negotiation.cc:240] Negotiation successful
0808 18:30:18.174013 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"thread_start_us":202,"threads_started":1}
I20260808 18:30:18.176213 10489 heartbeater.cc:347] Connected to a master server at 127.29.78.190:42713
I20260808 18:30:18.176527 10489 heartbeater.cc:464] Registering TS with master...
I20260808 18:30:18.177107 10489 heartbeater.cc:511] Master 127.29.78.190:42713 requested a full tablet report, sending...
I20260808 18:30:18.178190 9955 ts_manager.cc:194] Registered new tserver with Master: 027224b7aa174cc4a7f0dc4ee97b7645 (127.29.78.131:38015)
I20260808 18:30:18.178932 9955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:38403
I20260808 18:30:18.180960 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:30:18.192788 10193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:30:18.183452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55558 (local address 127.29.78.190:42713)
0808 18:30:18.183570 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 18:30:18.183574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:30:18.183930 (+ 356us) server_negotiation.cc:407] Connection header received
0808 18:30:18.184037 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:30:18.184041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:30:18.184088 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:30:18.184175 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:30:18.185042 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.185820 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.186995 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.187319 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.187434 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:30:18.189067 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:30:18.189071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:30:18.189074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:30:18.189101 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:30:18.190911 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:30:18.191420 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:30:18.191424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:30:18.191425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:30:18.191513 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:30:18.191816 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:30:18.191820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:30:18.191821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:30:18.192047 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:30:18.192187 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 18:30:18.192509 (+ 322us) server_negotiation.cc:299] Negotiation successful
0808 18:30:18.192633 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 18:30:18.196192 9955 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:30:18.200527 9955 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:30:18.222641 10507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:30:18.210538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:39375 (local address 127.0.0.1:38820)
0808 18:30:18.215063 (+ 4525us) negotiation.cc:107] Waiting for socket to connect
0808 18:30:18.215276 (+ 213us) client_negotiation.cc:174] Beginning negotiation
0808 18:30:18.215412 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:30:18.216426 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:30:18.216434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:30:18.216476 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:30:18.216630 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:30:18.216639 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.218665 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.218670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:30:18.220251 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:30:18.220265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.222019 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.222022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:30:18.222151 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:30:18.222198 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 18:30:18.222309 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 18:30:18.222451 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4341,"thread_start_us":110,"threads_started":1}
I20260808 18:30:18.227524 10505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:30:18.211250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38820 (local address 127.29.78.130:39375)
0808 18:30:18.215722 (+ 4472us) server_negotiation.cc:206] Beginning negotiation
0808 18:30:18.215736 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 18:30:18.215762 (+ 26us) server_negotiation.cc:407] Connection header received
0808 18:30:18.215844 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:30:18.215849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:30:18.216071 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:30:18.216231 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:30:18.217088 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.218525 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.220448 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.221348 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.221470 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:30:18.221623 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 18:30:18.227250 (+ 5627us) server_negotiation.cc:299] Negotiation successful
0808 18:30:18.227375 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4364,"thread_start_us":240,"threads_started":1}
I20260808 18:30:18.228976 10508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:30:18.213042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38836 (local address 127.29.78.129:46255)
0808 18:30:18.214904 (+ 1862us) server_negotiation.cc:206] Beginning negotiation
0808 18:30:18.214912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 18:30:18.217696 (+ 2784us) server_negotiation.cc:407] Connection header received
0808 18:30:18.217778 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:30:18.217783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:30:18.217962 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:30:18.218102 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:30:18.218542 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.221256 (+ 2714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.226045 (+ 4789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.226908 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.227018 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:30:18.227167 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 18:30:18.228741 (+ 1574us) server_negotiation.cc:299] Negotiation successful
0808 18:30:18.228839 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1745,"thread_start_us":127,"threads_started":1}
I20260808 18:30:18.229336 10509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:30:18.212231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:46255 (local address 127.0.0.1:38836)
0808 18:30:18.215063 (+ 2832us) negotiation.cc:107] Waiting for socket to connect
0808 18:30:18.217497 (+ 2434us) client_negotiation.cc:174] Beginning negotiation
0808 18:30:18.217593 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:30:18.218245 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:30:18.218248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:30:18.218264 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:30:18.218374 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:30:18.218380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.221400 (+ 3020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.221403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:30:18.225897 (+ 4494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:30:18.225908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.227727 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.227730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:30:18.227827 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:30:18.227856 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 18:30:18.227920 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 18:30:18.229177 (+ 1257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2753,"mutex_wait_us":1406,"thread_start_us":240,"threads_started":1}
I20260808 18:30:18.232079 10504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:30:18.213841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:38015 (local address 127.0.0.1:52700)
0808 18:30:18.216870 (+ 3029us) negotiation.cc:107] Waiting for socket to connect
0808 18:30:18.216885 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:30:18.216967 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:30:18.221683 (+ 4716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:30:18.221686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:30:18.221708 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:30:18.221887 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:30:18.221896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.229510 (+ 7614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.229514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:30:18.230617 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:30:18.230627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.231715 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.231719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:30:18.231817 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:30:18.231840 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 18:30:18.231889 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 18:30:18.231951 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2955,"mutex_wait_us":1,"thread_start_us":76,"threads_started":1}
I20260808 18:30:18.232833 10510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:30:18.214238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52700 (local address 127.29.78.131:38015)
0808 18:30:18.220451 (+ 6213us) server_negotiation.cc:206] Beginning negotiation
0808 18:30:18.220464 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 18:30:18.220490 (+ 26us) server_negotiation.cc:407] Connection header received
0808 18:30:18.220568 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:30:18.220573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:30:18.220742 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:30:18.220887 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:30:18.224153 (+ 3266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.225330 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.230790 (+ 5460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.231571 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.232405 (+ 834us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:30:18.232554 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 18:30:18.232604 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 18:30:18.232692 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6100,"thread_start_us":306,"threads_started":1}
I20260808 18:30:18.233412 10105 tablet_service.cc:1467] Processing CreateTablet for tablet 170d93fd18bb4790b745eb7cb3ffb1d1 (DEFAULT_TABLE table=test-table [id=aca9106c3ad94c5b848373b90370f8fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:30:18.235386 10267 tablet_service.cc:1467] Processing CreateTablet for tablet 170d93fd18bb4790b745eb7cb3ffb1d1 (DEFAULT_TABLE table=test-table [id=aca9106c3ad94c5b848373b90370f8fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:30:18.235386 10397 tablet_service.cc:1467] Processing CreateTablet for tablet 170d93fd18bb4790b745eb7cb3ffb1d1 (DEFAULT_TABLE table=test-table [id=aca9106c3ad94c5b848373b90370f8fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:30:18.236433 10267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 170d93fd18bb4790b745eb7cb3ffb1d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:30:18.236433 10397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 170d93fd18bb4790b745eb7cb3ffb1d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:30:18.238857 10105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 170d93fd18bb4790b745eb7cb3ffb1d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:30:18.246966 10512 tablet_bootstrap.cc:492] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd: Bootstrap starting.
I20260808 18:30:18.249347 10512 tablet_bootstrap.cc:654] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd: Neither blocks nor log segments found. Creating new log.
I20260808 18:30:18.250345 10512 log.cc:824] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd: Log is configured to *not* fsync() on all Append() calls
I20260808 18:30:18.251332 10511 tablet_bootstrap.cc:492] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 027224b7aa174cc4a7f0dc4ee97b7645: Bootstrap starting.
I20260808 18:30:18.253049 10514 tablet_bootstrap.cc:492] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 4210311bae3742e1887b08b1b940aefe: Bootstrap starting.
I20260808 18:30:18.254680 10511 tablet_bootstrap.cc:654] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 027224b7aa174cc4a7f0dc4ee97b7645: Neither blocks nor log segments found. Creating new log.
I20260808 18:30:18.255515 10511 log.cc:824] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 027224b7aa174cc4a7f0dc4ee97b7645: Log is configured to *not* fsync() on all Append() calls
I20260808 18:30:18.256740 10514 tablet_bootstrap.cc:654] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 4210311bae3742e1887b08b1b940aefe: Neither blocks nor log segments found. Creating new log.
I20260808 18:30:18.257558 10514 log.cc:824] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 4210311bae3742e1887b08b1b940aefe: Log is configured to *not* fsync() on all Append() calls
I20260808 18:30:18.257795 10512 tablet_bootstrap.cc:492] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd: No bootstrap required, opened a new log
I20260808 18:30:18.258023 10512 ts_tablet_manager.cc:1397] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260808 18:30:18.259989 10511 tablet_bootstrap.cc:492] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 027224b7aa174cc4a7f0dc4ee97b7645: No bootstrap required, opened a new log
I20260808 18:30:18.261612 10514 tablet_bootstrap.cc:492] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 4210311bae3742e1887b08b1b940aefe: No bootstrap required, opened a new log
I20260808 18:30:18.261911 10514 ts_tablet_manager.cc:1397] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 4210311bae3742e1887b08b1b940aefe: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260808 18:30:18.262924 10512 raft_consensus.cc:348] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4210311bae3742e1887b08b1b940aefe" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39375 } } peers { permanent_uuid: "858a38791b62436dbccf8eb47d4848dd" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46255 } } peers { permanent_uuid: "027224b7aa174cc4a7f0dc4ee97b7645" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38015 } }
I20260808 18:30:18.263217 10512 raft_consensus.cc:374] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:30:18.263270 10512 raft_consensus.cc:729] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 858a38791b62436dbccf8eb47d4848dd, State: Initialized, Role: FOLLOWER
I20260808 18:30:18.263752 10512 consensus_queue.cc:260] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4210311bae3742e1887b08b1b940aefe" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39375 } } peers { permanent_uuid: "858a38791b62436dbccf8eb47d4848dd" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46255 } } peers { permanent_uuid: "027224b7aa174cc4a7f0dc4ee97b7645" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38015 } }
I20260808 18:30:18.264917 10512 ts_tablet_manager.cc:1428] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260808 18:30:18.265527 10190 heartbeater.cc:503] Master 127.29.78.190:42713 was elected leader, sending a full tablet report...
I20260808 18:30:18.266984 10514 raft_consensus.cc:348] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 4210311bae3742e1887b08b1b940aefe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4210311bae3742e1887b08b1b940aefe" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39375 } } peers { permanent_uuid: "858a38791b62436dbccf8eb47d4848dd" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46255 } } peers { permanent_uuid: "027224b7aa174cc4a7f0dc4ee97b7645" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38015 } }
I20260808 18:30:18.267272 10514 raft_consensus.cc:374] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 4210311bae3742e1887b08b1b940aefe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:30:18.267362 10514 raft_consensus.cc:729] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 4210311bae3742e1887b08b1b940aefe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4210311bae3742e1887b08b1b940aefe, State: Initialized, Role: FOLLOWER
I20260808 18:30:18.267788 10514 consensus_queue.cc:260] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 4210311bae3742e1887b08b1b940aefe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4210311bae3742e1887b08b1b940aefe" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39375 } } peers { permanent_uuid: "858a38791b62436dbccf8eb47d4848dd" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46255 } } peers { permanent_uuid: "027224b7aa174cc4a7f0dc4ee97b7645" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38015 } }
I20260808 18:30:18.269050 10514 ts_tablet_manager.cc:1428] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 4210311bae3742e1887b08b1b940aefe: Time spent starting tablet: real 0.007s user 0.000s sys 0.005s
I20260808 18:30:18.269413 10340 heartbeater.cc:503] Master 127.29.78.190:42713 was elected leader, sending a full tablet report...
I20260808 18:30:18.269644 10511 ts_tablet_manager.cc:1397] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 027224b7aa174cc4a7f0dc4ee97b7645: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.006s
I20260808 18:30:18.274593 10511 raft_consensus.cc:348] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 027224b7aa174cc4a7f0dc4ee97b7645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4210311bae3742e1887b08b1b940aefe" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39375 } } peers { permanent_uuid: "858a38791b62436dbccf8eb47d4848dd" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46255 } } peers { permanent_uuid: "027224b7aa174cc4a7f0dc4ee97b7645" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38015 } }
I20260808 18:30:18.274887 10511 raft_consensus.cc:374] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 027224b7aa174cc4a7f0dc4ee97b7645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:30:18.274983 10511 raft_consensus.cc:729] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 027224b7aa174cc4a7f0dc4ee97b7645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 027224b7aa174cc4a7f0dc4ee97b7645, State: Initialized, Role: FOLLOWER
I20260808 18:30:18.275454 10511 consensus_queue.cc:260] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 027224b7aa174cc4a7f0dc4ee97b7645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4210311bae3742e1887b08b1b940aefe" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39375 } } peers { permanent_uuid: "858a38791b62436dbccf8eb47d4848dd" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46255 } } peers { permanent_uuid: "027224b7aa174cc4a7f0dc4ee97b7645" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38015 } }
I20260808 18:30:18.276604 10511 ts_tablet_manager.cc:1428] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 027224b7aa174cc4a7f0dc4ee97b7645: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260808 18:30:18.276938 10489 heartbeater.cc:503] Master 127.29.78.190:42713 was elected leader, sending a full tablet report...
W20260808 18:30:18.281847 10341 tablet.cc:2369] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 4210311bae3742e1887b08b1b940aefe: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 18:30:18 dist-test-slave-gp6n krb5kdc[9430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213817, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260808 18:30:18.363229 10518 raft_consensus.cc:482] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:30:18.363524 10518 raft_consensus.cc:504] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4210311bae3742e1887b08b1b940aefe" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39375 } } peers { permanent_uuid: "858a38791b62436dbccf8eb47d4848dd" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46255 } } peers { permanent_uuid: "027224b7aa174cc4a7f0dc4ee97b7645" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38015 } }
I20260808 18:30:18.364759 10518 leader_election.cc:288] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4210311bae3742e1887b08b1b940aefe (127.29.78.130:39375), 027224b7aa174cc4a7f0dc4ee97b7645 (127.29.78.131:38015)
I20260808 18:30:18.371865 10505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:30:18.365463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:44177 (local address 127.29.78.130:39375)
0808 18:30:18.365612 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 18:30:18.365617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:30:18.365915 (+ 298us) server_negotiation.cc:407] Connection header received
0808 18:30:18.366133 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:30:18.366137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:30:18.366194 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:30:18.366245 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:30:18.366734 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.367405 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.370459 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.371070 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.371173 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:30:18.371227 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 18:30:18.371604 (+ 377us) server_negotiation.cc:299] Negotiation successful
0808 18:30:18.371694 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 18:30:18.373333 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:30:18.364944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:39375 (local address 127.29.78.129:44177)
0808 18:30:18.365824 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0808 18:30:18.365840 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:30:18.366001 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:30:18.366393 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:30:18.366396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:30:18.366440 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:30:18.366597 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:30:18.366604 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.367568 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.367571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:30:18.370304 (+ 2733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:30:18.370315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.371338 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.371342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:30:18.371445 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:30:18.371469 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 18:30:18.373086 (+ 1617us) client_negotiation.cc:240] Negotiation successful
0808 18:30:18.373164 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":216,"threads_started":1}
I20260808 18:30:18.374184 10291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170d93fd18bb4790b745eb7cb3ffb1d1" candidate_uuid: "858a38791b62436dbccf8eb47d4848dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4210311bae3742e1887b08b1b940aefe" is_pre_election: true
I20260808 18:30:18.374606 10291 raft_consensus.cc:2454] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 4210311bae3742e1887b08b1b940aefe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 858a38791b62436dbccf8eb47d4848dd in term 0.
I20260808 18:30:18.375154 10065 leader_election.cc:302] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4210311bae3742e1887b08b1b940aefe, 858a38791b62436dbccf8eb47d4848dd; no voters:
I20260808 18:30:18.375443 10518 raft_consensus.cc:2781] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:30:18.375536 10518 raft_consensus.cc:482] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:30:18.375588 10518 raft_consensus.cc:3037] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:30:18.376405 10510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:30:18.365665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:48061 (local address 127.29.78.131:38015)
0808 18:30:18.365795 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 18:30:18.365799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:30:18.368189 (+ 2390us) server_negotiation.cc:407] Connection header received
0808 18:30:18.368434 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:30:18.368438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:30:18.368507 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:30:18.368560 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:30:18.369988 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.370717 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.374039 (+ 3322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.374546 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.374648 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:30:18.374688 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0808 18:30:18.376208 (+ 1520us) server_negotiation.cc:299] Negotiation successful
0808 18:30:18.376291 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 18:30:18.376770 10524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:30:18.365181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:38015 (local address 127.29.78.129:48061)
0808 18:30:18.368081 (+ 2900us) negotiation.cc:107] Waiting for socket to connect
0808 18:30:18.368100 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 18:30:18.368265 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:30:18.369676 (+ 1411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:30:18.369680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:30:18.369702 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:30:18.369840 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:30:18.369847 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.371842 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.371845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:30:18.372987 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:30:18.372999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.375953 (+ 2954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.375956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:30:18.376059 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:30:18.376081 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 18:30:18.376547 (+ 466us) client_negotiation.cc:240] Negotiation successful
0808 18:30:18.376624 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2812,"thread_start_us":95,"threads_started":1}
I20260808 18:30:18.377153 10518 raft_consensus.cc:504] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4210311bae3742e1887b08b1b940aefe" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39375 } } peers { permanent_uuid: "858a38791b62436dbccf8eb47d4848dd" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46255 } } peers { permanent_uuid: "027224b7aa174cc4a7f0dc4ee97b7645" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38015 } }
I20260808 18:30:18.377523 10518 leader_election.cc:288] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd [CANDIDATE]: Term 1 election: Requested vote from peers 4210311bae3742e1887b08b1b940aefe (127.29.78.130:39375), 027224b7aa174cc4a7f0dc4ee97b7645 (127.29.78.131:38015)
I20260808 18:30:18.377540 10418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170d93fd18bb4790b745eb7cb3ffb1d1" candidate_uuid: "858a38791b62436dbccf8eb47d4848dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "027224b7aa174cc4a7f0dc4ee97b7645" is_pre_election: true
I20260808 18:30:18.377828 10418 raft_consensus.cc:2454] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 027224b7aa174cc4a7f0dc4ee97b7645 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 858a38791b62436dbccf8eb47d4848dd in term 0.
I20260808 18:30:18.378934 10418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170d93fd18bb4790b745eb7cb3ffb1d1" candidate_uuid: "858a38791b62436dbccf8eb47d4848dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "027224b7aa174cc4a7f0dc4ee97b7645"
I20260808 18:30:18.379065 10418 raft_consensus.cc:3037] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 027224b7aa174cc4a7f0dc4ee97b7645 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:30:18.380002 10291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "170d93fd18bb4790b745eb7cb3ffb1d1" candidate_uuid: "858a38791b62436dbccf8eb47d4848dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4210311bae3742e1887b08b1b940aefe"
I20260808 18:30:18.380141 10291 raft_consensus.cc:3037] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 4210311bae3742e1887b08b1b940aefe [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:30:18.380455 10418 raft_consensus.cc:2454] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 027224b7aa174cc4a7f0dc4ee97b7645 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 858a38791b62436dbccf8eb47d4848dd in term 1.
I20260808 18:30:18.380863 10063 leader_election.cc:302] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 027224b7aa174cc4a7f0dc4ee97b7645, 858a38791b62436dbccf8eb47d4848dd; no voters:
I20260808 18:30:18.381023 10518 raft_consensus.cc:2781] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:30:18.381247 10518 raft_consensus.cc:686] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd [term 1 LEADER]: Becoming Leader. State: Replica: 858a38791b62436dbccf8eb47d4848dd, State: Running, Role: LEADER
I20260808 18:30:18.381523 10291 raft_consensus.cc:2454] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 4210311bae3742e1887b08b1b940aefe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 858a38791b62436dbccf8eb47d4848dd in term 1.
I20260808 18:30:18.381628 10518 consensus_queue.cc:237] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4210311bae3742e1887b08b1b940aefe" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39375 } } peers { permanent_uuid: "858a38791b62436dbccf8eb47d4848dd" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46255 } } peers { permanent_uuid: "027224b7aa174cc4a7f0dc4ee97b7645" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38015 } }
I20260808 18:30:18.387900 9954 catalog_manager.cc:5495] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 858a38791b62436dbccf8eb47d4848dd (127.29.78.129). New cstate: current_term: 1 leader_uuid: "858a38791b62436dbccf8eb47d4848dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4210311bae3742e1887b08b1b940aefe" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "858a38791b62436dbccf8eb47d4848dd" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "027224b7aa174cc4a7f0dc4ee97b7645" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38015 } health_report { overall_health: UNKNOWN } } }
W20260808 18:30:18.408107 10490 tablet.cc:2369] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 027224b7aa174cc4a7f0dc4ee97b7645: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:30:18.416268 10508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:30:18.410578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38840 (local address 127.29.78.129:46255)
0808 18:30:18.410711 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 18:30:18.410716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:30:18.410730 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:30:18.410827 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:30:18.410830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:30:18.410898 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:30:18.411025 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:30:18.411604 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.412457 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.413985 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:30:18.414335 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:30:18.414473 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:30:18.414915 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:30:18.415501 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:30:18.415621 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 18:30:18.416052 (+ 431us) server_negotiation.cc:299] Negotiation successful
0808 18:30:18.416122 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 18:30:18.424045 10418 raft_consensus.cc:1261] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 027224b7aa174cc4a7f0dc4ee97b7645 [term 1 FOLLOWER]: Refusing update from remote peer 858a38791b62436dbccf8eb47d4848dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:30:18.424904 10518 consensus_queue.cc:1035] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "027224b7aa174cc4a7f0dc4ee97b7645" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:30:18.432452 10291 raft_consensus.cc:1261] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 4210311bae3742e1887b08b1b940aefe [term 1 FOLLOWER]: Refusing update from remote peer 858a38791b62436dbccf8eb47d4848dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:30:18.433197 10518 consensus_queue.cc:1035] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 858a38791b62436dbccf8eb47d4848dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4210311bae3742e1887b08b1b940aefe" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:30:18.447505 10529 mvcc.cc:205] Tried to move back new op lower bound from 7316331800255709184 to 7316331800096104448. Current Snapshot: MvccSnapshot[applied={T|T < 7316331800255709184}]
I20260808 18:30:18.448796 10534 mvcc.cc:205] Tried to move back new op lower bound from 7316331800255709184 to 7316331800096104448. Current Snapshot: MvccSnapshot[applied={T|T < 7316331800255709184}]
I20260808 18:30:18.456354 10531 mvcc.cc:205] Tried to move back new op lower bound from 7316331800255709184 to 7316331800096104448. Current Snapshot: MvccSnapshot[applied={T|T < 7316331800255709184}]
I20260808 18:30:18.462766 9954 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:30:18.462939 9954 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:30:18.465739 9954 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b59d9fde4d574db0970ac3fdc051127f: Sending DeleteTablet for 3 replicas of tablet 170d93fd18bb4790b745eb7cb3ffb1d1
I20260808 18:30:18.466866 10105 tablet_service.cc:1514] Processing DeleteTablet for tablet 170d93fd18bb4790b745eb7cb3ffb1d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:30:18 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:38836
I20260808 18:30:18.466890 10397 tablet_service.cc:1514] Processing DeleteTablet for tablet 170d93fd18bb4790b745eb7cb3ffb1d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:30:18 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:52700
I20260808 18:30:18.467620 10542 tablet_replica.cc:330] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 027224b7aa174cc4a7f0dc4ee97b7645: stopping tablet replica
I20260808 18:30:18.467837 10267 tablet_service.cc:1514] Processing DeleteTablet for tablet 170d93fd18bb4790b745eb7cb3ffb1d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:30:18 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:38820
I20260808 18:30:18.468027 10542 raft_consensus.cc:2229] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 027224b7aa174cc4a7f0dc4ee97b7645 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:30:18.468389 10542 raft_consensus.cc:2258] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 027224b7aa174cc4a7f0dc4ee97b7645 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:30:18.469063 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 10044
I20260808 18:30:18.474436 10542 ts_tablet_manager.cc:1905] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 027224b7aa174cc4a7f0dc4ee97b7645: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:30:18.477628 10542 ts_tablet_manager.cc:1918] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 027224b7aa174cc4a7f0dc4ee97b7645: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:30:18.477747 10542 log.cc:1196] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 027224b7aa174cc4a7f0dc4ee97b7645: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/170d93fd18bb4790b745eb7cb3ffb1d1
I20260808 18:30:18.478139 10542 ts_tablet_manager.cc:1939] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 027224b7aa174cc4a7f0dc4ee97b7645: Deleting consensus metadata
I20260808 18:30:18.479737 9938 catalog_manager.cc:4841] TS 027224b7aa174cc4a7f0dc4ee97b7645 (127.29.78.131:38015): tablet 170d93fd18bb4790b745eb7cb3ffb1d1 (table test-table [id=aca9106c3ad94c5b848373b90370f8fb]) successfully deleted
I20260808 18:30:18.483009 10543 tablet_replica.cc:330] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 4210311bae3742e1887b08b1b940aefe: stopping tablet replica
I20260808 18:30:18.483517 10543 raft_consensus.cc:2229] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 4210311bae3742e1887b08b1b940aefe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:30:18.484169 10543 raft_consensus.cc:2258] T 170d93fd18bb4790b745eb7cb3ffb1d1 P 4210311bae3742e1887b08b1b940aefe [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 18:30:18.484673 9939 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:46255 (error 108)
I20260808 18:30:18.485737 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 10196
I20260808 18:30:18.487540 10504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:30:18.487202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:46255 (local address 127.0.0.1:38842)
0808 18:30:18.487381 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 18:30:18.487455 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:46255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260808 18:30:18.487730 9939 catalog_manager.cc:4584] TS 858a38791b62436dbccf8eb47d4848dd (127.29.78.129:46255): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 170d93fd18bb4790b745eb7cb3ffb1d1: Network error: Client connection negotiation failed: client connection to 127.29.78.129:46255: connect: Connection refused (error 111)
W20260808 18:30:18.499434 10504 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:30:18.497415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:39375 (local address 127.0.0.1:38832)
0808 18:30:18.497614 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0808 18:30:18.497631 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:30:18.497737 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:30:18.499296 (+ 1559us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.130:39375: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":116}
I20260808 18:30:18.500675 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 10345
I20260808 18:30:18.532052 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 9920
I20260808 18:30:18.541103 30010 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 18:30:18.895836 30010 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 18:30:18.895972 30010 mini_ranger.cc:66] Stopping Ranger...
I20260808 18:30:19.240851 30010 mini_ranger.cc:68] Stopped Ranger
2026-08-08 18:30:19.241 UTC [9463] LOG: received smart shutdown request
2026-08-08 18:30:19.245 UTC [9463] LOG: background worker "logical replication launcher" (PID 9474) exited with exit code 1
2026-08-08 18:30:19.245 UTC [9468] LOG: shutting down
2026-08-08 18:30:19.426 UTC [9463] LOG: database system is shut down
2026-08-08T18:30:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47636 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:30:19 dist-test-slave-gp6n krb5kdc[10555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:30:19 dist-test-slave-gp6n krb5kdc[10555](info): set up 2 sockets
Aug 08 18:30:19 dist-test-slave-gp6n krb5kdc[10555](info): commencing operation
krb5kdc: starting...
W20260808 18:30:20.917078 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.346s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:30:21 dist-test-slave-gp6n krb5kdc[10555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:30:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:30:21Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.29.78.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.29.78.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/rangeradmin_127.29.78.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/rangeradmin_127.29.78.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.29.78.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.29.78.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/rangerlookup_127.29.78.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/rangerlookup_127.29.78.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.78.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.29.78.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/HTTP_127.29.78.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/HTTP_127.29.78.148@KRBTEST.COM.keytab.
I20260808 18:30:21.183007 30010 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task0kHl76/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/postgres -l logfile start
2026-08-08 18:30:23.822 UTC [10591] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 18:30:23.822 UTC [10591] LOG: listening on IPv4 address "127.29.78.148", port 44719
2026-08-08 18:30:23.826 UTC [10591] LOG: listening on Unix socket "/tmp/.s.PGSQL.44719"
2026-08-08 18:30:23.862 UTC [10594] LOG: database system was shut down at 2026-08-08 18:30:22 UTC
2026-08-08 18:30:23.866 UTC [10591] LOG: database system is ready to accept connections
I20260808 18:30:24.890777 30010 mini_postgres.cc:96] Postgres bound to 44719
2026-08-08 18:30:24.897 UTC [10605] FATAL: database "slave" does not exist
127.29.78.148:44719 - accepting connections
I20260808 18:30:24.898527 30010 mini_ranger.cc:161] Starting Ranger...
I20260808 18:30:24.914566 30010 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 18:30:25.413609 30010 mini_ranger.cc:87] Created ranger Postgres database
I20260808 18:30:25.413705 30010 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ranger-admin
2026-08-08 18:30:25,698 [I] DB FLAVOR :POSTGRES
2026-08-08 18:30:25,699 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 18:30:25,699 [I] Checking connection..
2026-08-08 18:30:25,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 18:30:25,951 [I] Checking connection passed.
2026-08-08 18:30:25,952 [I] --------- Verifying version history table ---------
2026-08-08 18:30:25,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 18:30:26,220 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 18:30:26,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 18:30:26,482 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 18:30:26,483 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 18:30:26,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 18:30:26.752 UTC [10678] WARNING: there is no transaction in progress
2026-08-08 18:30:26,759 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 18:30:26,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 18:30:26,997 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 18:30:26,998 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 18:30:26,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 18:30:27,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 18:30:27,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gp6n.c.gcp-upstream.internal','N') ;"
2026-08-08 18:30:27,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 18:30:28,021 [I] Table x_portal_user does not exist in database ranger
2026-08-08 18:30:28,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 18:30:28,275 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 18:30:28,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 18:30:28,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 18:30:28,748 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 18:30:28,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 18:30:29.745 UTC [10800] WARNING: there is no transaction in progress
2026-08-08 18:30:29.761 UTC [10800] WARNING: there is no transaction in progress
2026-08-08 18:30:29.777 UTC [10800] WARNING: there is no transaction in progress
2026-08-08 18:30:29.793 UTC [10800] WARNING: there is no transaction in progress
2026-08-08 18:30:29.812 UTC [10800] WARNING: there is no transaction in progress
2026-08-08 18:30:29.829 UTC [10800] WARNING: there is no transaction in progress
2026-08-08 18:30:29.887 UTC [10800] WARNING: there is no transaction in progress
2026-08-08 18:30:29.894 UTC [10800] WARNING: there is no transaction in progress
2026-08-08 18:30:29.908 UTC [10800] WARNING: there is no transaction in progress
2026-08-08 18:30:29.918 UTC [10800] WARNING: there is no transaction in progress
2026-08-08 18:30:29.929 UTC [10800] WARNING: there is no transaction in progress
2026-08-08 18:30:29.943 UTC [10800] WARNING: there is no transaction in progress
2026-08-08 18:30:29.953 UTC [10800] WARNING: there is no transaction in progress
2026-08-08 18:30:29.959 UTC [10800] WARNING: there is no transaction in progress
2026-08-08 18:30:29.966 UTC [10800] WARNING: there is no transaction in progress
2026-08-08 18:30:29.973 UTC [10800] WARNING: there is no transaction in progress
2026-08-08 18:30:30,636 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 18:30:30,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 18:30:30,884 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 18:30:30,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 18:30:31,149 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 18:30:31,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 18:30:31,412 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 18:30:31,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 18:30:31,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 18:30:31,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gp6n.c.gcp-upstream.internal';"
2026-08-08 18:30:32,184 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 18:30:32,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 18:30:32,427 [I] DB_PATCHES have already been applied
I20260808 18:30:32.433382 30010 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ranger-admin:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task0kHl76/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 18:30:32.433445 30010 mini_ranger.cc:193] Using host: 127.29.78.148
I20260808 18:30:32.437714 30010 mini_ranger.cc:227] Ranger admin URL: http://127.29.78.148:49969
Aug 08, 2026 6:30:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 6:30:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 6:30:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 6:30:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 6:30:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 6:30:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 6:30:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 18:30:33.600486 30010 mini_ranger.cc:160] Time spent starting Ranger: real 8.702s user 0.001s sys 0.011s
Aug 08, 2026 6:30:54 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 6:30:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 6:30:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 6:30:54 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 6:30:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 6:30:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 6:30:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 6:30:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 6:30:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 6:30:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 6:30:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 6:30:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 6:30:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 6:30:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 6:30:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 6:30:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 6:30:54 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 6:30:55 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 18:30:56.008612 30010 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.29.78.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.29.78.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/rangerkms_127.29.78.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/rangerkms_127.29.78.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.78.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.29.78.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.78.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/HTTP_127.29.78.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.78.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/HTTP_127.29.78.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 18:30:56 dist-test-slave-gp6n krb5kdc[10555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213856, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 18:30:56.099282 30010 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 18:30:56.110054 30010 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 18:30:57.011269 30010 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 18:30:57.011406 30010 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ranger-kms
I20260808 18:30:57.011436 30010 mini_ranger_kms.cc:227] Using postgres at 127.29.78.148:44719
2026-08-08 18:30:57,191 [I] DB FLAVOR :POSTGRES
2026-08-08 18:30:57,191 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 18:30:57,191 [I] Checking connection
2026-08-08 18:30:57,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 18:30:57,428 [I] connection success
2026-08-08 18:30:57,428 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 18:30:57,428 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 18:30:57,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 18:30:57,692 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 18:30:57,692 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 18:30:57,692 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 18:30:57,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:44719/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 18:30:58,007 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 18:30:58.492288 30010 mini_ranger_kms.cc:310] Created kms service
I20260808 18:30:58.671260 30010 mini_ranger_kms.cc:326] Created kudu user
I20260808 18:30:58.716848 30010 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 18:30:58 dist-test-slave-gp6n krb5kdc[10555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213856, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260808 18:30:59.142592 30010 mini_ranger_kms.cc:384] Added ranger policy
I20260808 18:30:59.142750 30010 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ranger-kms:/tmp/dist-test-task0kHl76/build/debug/bin/postgresql.jar:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task0kHl76/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 18:30:59.142772 30010 mini_ranger_kms.cc:240] Using host: 127.29.78.148
I20260808 18:30:59.147651 30010 mini_ranger_kms.cc:276] Ranger KMS PID: 11000
I20260808 18:30:59.147739 30010 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.29.78.148:54763
Aug 08, 2026 6:31:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 6:31:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 6:31:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 6:31:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 6:31:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 6:31:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 18:31:01 dist-test-slave-gp6n krb5kdc[10555](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786213861, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.78.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:01.562260 30010 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.463s user 0.023s sys 0.142s
I20260808 18:31:01.562371 30010 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 18:31:01.562495 30010 mini_ranger_kms.cc:401] 127.29.78.148:54763/kms/v1/keys
Aug 08 18:31:01 dist-test-slave-gp6n krb5kdc[10555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213861, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.78.148@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
Aug 08 18:31:02 dist-test-slave-gp6n krb5kdc[10555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213861, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.78.148@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
Aug 08 18:31:04 dist-test-slave-gp6n krb5kdc[10555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:31:04.670833 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:43957
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:40055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:43957
--ranger_config_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:54763/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:31:04.821110 11056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:04.821412 11056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:04.821491 11056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:04.825516 11056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:31:04.825603 11056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:04.825644 11056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:04.825698 11056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:31:04.825734 11056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:31:04.831449 11056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:40055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/wal
--ranger_kms_url=127.29.78.148:54763/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:43957
--ranger_config_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:43957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:04.832530 11056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:04.833570 11056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:31:04.842798 11056 server_base.cc:1034] running on GCE node
W20260808 18:31:04.842767 11062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:04.842775 11061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:04.843185 11064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:04.843881 11056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:04.845269 11056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:04.846469 11056 hybrid_clock.cc:648] HybridClock initialized: now 1786213864846447 us; error 40 us; skew 500 ppm
Aug 08 18:31:04 dist-test-slave-gp6n krb5kdc[10555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213864, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:04.849936 11056 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:31:04.851336 11056 webserver.cc:460] Webserver started at http://127.29.78.190:38445/ using document root <none> and password file <none>
I20260808 18:31:04.851890 11056 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:04.851948 11056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:04.852171 11056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 18:31:04 dist-test-slave-gp6n krb5kdc[10555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213864, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260808 18:31:04.900985 11056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "c6fb0b1f5e5c4e36bd62e3e657dc4ee6"
format_stamp: "Formatted at 2026-08-08 18:31:04 on dist-test-slave-gp6n"
server_key: "be35fb7dd3b158fc4b09c481b2a07f3e"
server_key_iv: "e4007e12861dd1b824bcb2e4588d2832"
server_key_version: "kuduclusterkey@0"
I20260808 18:31:04.901685 11056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "c6fb0b1f5e5c4e36bd62e3e657dc4ee6"
format_stamp: "Formatted at 2026-08-08 18:31:04 on dist-test-slave-gp6n"
server_key: "be35fb7dd3b158fc4b09c481b2a07f3e"
server_key_iv: "e4007e12861dd1b824bcb2e4588d2832"
server_key_version: "kuduclusterkey@0"
I20260808 18:31:04.905915 11056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260808 18:31:04.920748 11071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:04.922235 11056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:31:04.922348 11056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "c6fb0b1f5e5c4e36bd62e3e657dc4ee6"
format_stamp: "Formatted at 2026-08-08 18:31:04 on dist-test-slave-gp6n"
server_key: "be35fb7dd3b158fc4b09c481b2a07f3e"
server_key_iv: "e4007e12861dd1b824bcb2e4588d2832"
server_key_version: "kuduclusterkey@0"
I20260808 18:31:04.922544 11056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:04.944029 11056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:04.947273 11056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:04.947499 11056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:04.956161 11056 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:43957
I20260808 18:31:04.956168 11129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:43957 every 8 connection(s)
I20260808 18:31:04.957162 11056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:31:04.960129 11130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:31:04.960276 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 11056
I20260808 18:31:04.960412 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:31:04.960739 30010 external_mini_cluster.cc:1428] Setting key 941fd157f99b72d66123eeab988a5514
I20260808 18:31:04.967519 11130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6: Bootstrap starting.
I20260808 18:31:04.970162 11130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6: Neither blocks nor log segments found. Creating new log.
I20260808 18:31:04.971125 11130 log.cc:824] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6: Log is configured to *not* fsync() on all Append() calls
Aug 08 18:31:04 dist-test-slave-gp6n krb5kdc[10555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:31:04.973433 11130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6: No bootstrap required, opened a new log
I20260808 18:31:04.977602 11130 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6fb0b1f5e5c4e36bd62e3e657dc4ee6" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43957 } }
I20260808 18:31:04.977897 11130 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:31:04.978005 11130 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6fb0b1f5e5c4e36bd62e3e657dc4ee6, State: Initialized, Role: FOLLOWER
I20260808 18:31:04.978370 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:04.962218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41584 (local address 127.29.78.190:43957)
0808 18:31:04.962686 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:04.962704 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:04.962749 (+ 45us) server_negotiation.cc:407] Connection header received
0808 18:31:04.963763 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:04.963787 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:04.964208 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:04.964595 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:04.965490 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:04.966993 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:04.968233 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:04.968935 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:04.969099 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:04.972230 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:04.972236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:04.972251 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:04.972280 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:04.975145 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:04.975655 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:04.975661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:04.975667 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:04.975803 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:04.976116 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:04.976119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:04.976121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:04.976638 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:04.976839 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:04.977153 (+ 314us) server_negotiation.cc:299] Negotiation successful
0808 18:31:04.977424 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":164,"threads_started":1}
I20260808 18:31:04.978503 11130 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6fb0b1f5e5c4e36bd62e3e657dc4ee6" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43957 } }
I20260808 18:31:04.978681 11130 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:31:04.978754 11130 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:31:04.978853 11130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:31:04.979894 11130 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6fb0b1f5e5c4e36bd62e3e657dc4ee6" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43957 } }
I20260808 18:31:04.980257 11130 leader_election.cc:302] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6fb0b1f5e5c4e36bd62e3e657dc4ee6; no voters:
I20260808 18:31:04.980640 11130 leader_election.cc:288] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:31:04.980918 11135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:31:04.981204 11135 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [term 1 LEADER]: Becoming Leader. State: Replica: c6fb0b1f5e5c4e36bd62e3e657dc4ee6, State: Running, Role: LEADER
I20260808 18:31:04.981657 11135 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6fb0b1f5e5c4e36bd62e3e657dc4ee6" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43957 } }
I20260808 18:31:04.981986 11130 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:31:04.983426 11136 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6fb0b1f5e5c4e36bd62e3e657dc4ee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6fb0b1f5e5c4e36bd62e3e657dc4ee6" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43957 } } }
I20260808 18:31:04.983546 11136 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [sys.catalog]: This master's current role is: LEADER
I20260808 18:31:04.983805 11137 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6fb0b1f5e5c4e36bd62e3e657dc4ee6. Latest consensus state: current_term: 1 leader_uuid: "c6fb0b1f5e5c4e36bd62e3e657dc4ee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6fb0b1f5e5c4e36bd62e3e657dc4ee6" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43957 } } }
I20260808 18:31:04.983919 11137 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [sys.catalog]: This master's current role is: LEADER
I20260808 18:31:04.984035 11150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:31:04.984371 11130 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 18:31:04.990078 11150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:31:05.000263 11150 catalog_manager.cc:1347] Generated new cluster ID: 1cec2c3ced0d43a7828504823d1d83a2
I20260808 18:31:05.000337 11150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:31:05.025200 11150 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:31:05.026664 11150 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:31:05.036453 11150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6: Generated new TSK 0
I20260808 18:31:05.037359 11150 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260808 18:31:05.811591 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:54763/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43957
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:40055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:31:05.935607 11183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:05.935882 11183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:05.935963 11183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:05.940055 11183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:05.940138 11183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:05.940237 11183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:31:05.945521 11183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:40055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.29.78.148:54763/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:05.946657 11183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:05.947676 11183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:05.957585 11188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:05.957998 11189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:05.958551 11191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:05.959731 11183 server_base.cc:1034] running on GCE node
I20260808 18:31:05.960139 11183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:05.960817 11183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:05.961993 11183 hybrid_clock.cc:648] HybridClock initialized: now 1786213865961965 us; error 42 us; skew 500 ppm
Aug 08 18:31:05 dist-test-slave-gp6n krb5kdc[10555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213865, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:05.965775 11183 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:31:05.967001 11183 webserver.cc:460] Webserver started at http://127.29.78.129:39271/ using document root <none> and password file <none>
I20260808 18:31:05.967774 11183 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:05.967887 11183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:05.968170 11183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 18:31:05 dist-test-slave-gp6n krb5kdc[10555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213865, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260808 18:31:05.985605 11183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "d57e1e0961384085b45cc5d104b2b381"
format_stamp: "Formatted at 2026-08-08 18:31:05 on dist-test-slave-gp6n"
server_key: "c98edee75ef7e8d3659b9ab7b7041517"
server_key_iv: "14548cbbaf07183c3db4d3a86619666e"
server_key_version: "kuduclusterkey@0"
I20260808 18:31:05.986315 11183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "d57e1e0961384085b45cc5d104b2b381"
format_stamp: "Formatted at 2026-08-08 18:31:05 on dist-test-slave-gp6n"
server_key: "c98edee75ef7e8d3659b9ab7b7041517"
server_key_iv: "14548cbbaf07183c3db4d3a86619666e"
server_key_version: "kuduclusterkey@0"
I20260808 18:31:05.992242 11183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 18:31:06.010100 11201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:06.012070 11183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:31:06.012226 11183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "d57e1e0961384085b45cc5d104b2b381"
format_stamp: "Formatted at 2026-08-08 18:31:05 on dist-test-slave-gp6n"
server_key: "c98edee75ef7e8d3659b9ab7b7041517"
server_key_iv: "14548cbbaf07183c3db4d3a86619666e"
server_key_version: "kuduclusterkey@0"
I20260808 18:31:06.012384 11183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:06.037879 11183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:06.042048 11183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:06.042333 11183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:06.043232 11183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:31:06.044472 11183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:31:06.044544 11183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:06.044615 11183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:31:06.044696 11183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260808 18:31:06.056409 11183 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:41487
I20260808 18:31:06.057758 11183 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:31:06.064893 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 11183
I20260808 18:31:06.065033 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:31:06.065392 30010 external_mini_cluster.cc:1428] Setting key e3a4f4cd74ddc2f94fb1b09d9d2e3f3d
I20260808 18:31:06.066469 11328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:41487 every 8 connection(s)
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
Aug 08 18:31:06 dist-test-slave-gp6n krb5kdc[10555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213865, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
I20260808 18:31:06.092274 11331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:06.066982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:48737 (local address 127.29.78.190:43957)
0808 18:31:06.073300 (+ 6318us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:06.073306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:06.074802 (+ 1496us) server_negotiation.cc:407] Connection header received
0808 18:31:06.075328 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:06.075333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:06.075399 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:06.075496 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:06.077174 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:06.078085 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:06.079851 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:06.080250 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:06.080333 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:06.087477 (+ 7144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:06.087481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:06.087484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:06.087519 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:06.089565 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:06.090254 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:06.090258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:06.090260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:06.090355 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:06.090842 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:06.090846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:06.090848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:06.091105 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:06.091192 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:06.091912 (+ 720us) server_negotiation.cc:299] Negotiation successful
0808 18:31:06.092072 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6218,"thread_start_us":85,"threads_started":1}
I20260808 18:31:06.094435 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:06.067495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43957 (local address 127.29.78.129:48737)
0808 18:31:06.073563 (+ 6068us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:06.073632 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:06.074928 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:06.075843 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:06.075853 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:06.076254 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:06.076968 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:06.077003 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:06.078251 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:06.078255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:06.079701 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:06.079713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:06.080559 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:06.080563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:06.080777 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:06.081532 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:06.081556 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:06.084722 (+ 3166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:06.089715 (+ 4993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:06.089722 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:06.089743 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:06.090127 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:06.090523 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:06.090526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:06.090528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:06.090727 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:06.091260 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:06.091266 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:06.091613 (+ 347us) client_negotiation.cc:769] Sending connection context
0808 18:31:06.092765 (+ 1152us) client_negotiation.cc:240] Negotiation successful
0808 18:31:06.093170 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5793,"thread_start_us":181,"threads_started":1}
I20260808 18:31:06.096215 11329 heartbeater.cc:347] Connected to a master server at 127.29.78.190:43957
I20260808 18:31:06.096482 11329 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:06.097150 11329 heartbeater.cc:511] Master 127.29.78.190:43957 requested a full tablet report, sending...
I20260808 18:31:06.099171 11088 ts_manager.cc:194] Registered new tserver with Master: d57e1e0961384085b45cc5d104b2b381 (127.29.78.129:41487)
I20260808 18:31:06.100821 11088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:48737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260808 18:31:06.125322 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:54763/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43957
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:40055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 18:31:06 dist-test-slave-gp6n krb5kdc[10555](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786213866, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 18:31:06.300877 11335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:06.301368 11335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:06.301508 11335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:06.308502 11335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:06.308678 11335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:06.308851 11335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:31:06.317795 11335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:40055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.29.78.148:54763/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:06.319211 11335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:06.320613 11335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:06.335891 11342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:06.336098 11343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:06.336819 11335 server_base.cc:1034] running on GCE node
W20260808 18:31:06.338572 11345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:06.339172 11335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:06.340051 11335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:06.341264 11335 hybrid_clock.cc:648] HybridClock initialized: now 1786213866341238 us; error 39 us; skew 500 ppm
Aug 08 18:31:06 dist-test-slave-gp6n krb5kdc[10555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213866, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:06.345288 11335 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:31:06.346483 11335 webserver.cc:460] Webserver started at http://127.29.78.130:32979/ using document root <none> and password file <none>
I20260808 18:31:06.347218 11335 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:06.347323 11335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:06.347558 11335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 18:31:06 dist-test-slave-gp6n krb5kdc[10555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213866, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260808 18:31:06.369843 11335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "a4279ac13fc44ed2a1e345cc78656e7e"
format_stamp: "Formatted at 2026-08-08 18:31:06 on dist-test-slave-gp6n"
server_key: "cbce267690434a04a4255d5688d43fde"
server_key_iv: "eb83412e40877433de899e337c64cdb4"
server_key_version: "kuduclusterkey@0"
I20260808 18:31:06.371121 11335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "a4279ac13fc44ed2a1e345cc78656e7e"
format_stamp: "Formatted at 2026-08-08 18:31:06 on dist-test-slave-gp6n"
server_key: "cbce267690434a04a4255d5688d43fde"
server_key_iv: "eb83412e40877433de899e337c64cdb4"
server_key_version: "kuduclusterkey@0"
I20260808 18:31:06.377041 11335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 18:31:06.400862 11354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:06.402845 11335 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260808 18:31:06.403013 11335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "a4279ac13fc44ed2a1e345cc78656e7e"
format_stamp: "Formatted at 2026-08-08 18:31:06 on dist-test-slave-gp6n"
server_key: "cbce267690434a04a4255d5688d43fde"
server_key_iv: "eb83412e40877433de899e337c64cdb4"
server_key_version: "kuduclusterkey@0"
I20260808 18:31:06.403175 11335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:06.450881 11335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:06.455098 11335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:06.455417 11335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:06.456254 11335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:31:06.457499 11335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:31:06.457628 11335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:06.457746 11335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:31:06.457823 11335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:06.473711 11335 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:34905
I20260808 18:31:06.475342 11335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:31:06.475564 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 11335
I20260808 18:31:06.475695 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:31:06.476046 30010 external_mini_cluster.cc:1428] Setting key e1e40c5cba69602e8e0f777ca2fe15f4
I20260808 18:31:06.479105 11480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:34905 every 8 connection(s)
Aug 08 18:31:06 dist-test-slave-gp6n krb5kdc[10555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213866, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:31:06.496011 11331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:06.480816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:60747 (local address 127.29.78.190:43957)
0808 18:31:06.480955 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:06.480960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:06.480978 (+ 18us) server_negotiation.cc:407] Connection header received
0808 18:31:06.482011 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:06.482015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:06.482076 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:06.482136 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:06.484039 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:06.484950 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:06.486903 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:06.487291 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:06.487406 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:06.490881 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:06.490886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:06.490890 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:06.490924 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:06.493071 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:06.493829 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:06.493834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:06.493835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:06.493936 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:06.494546 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:06.494550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:06.494552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:06.494809 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:06.494910 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:06.495653 (+ 743us) server_negotiation.cc:299] Negotiation successful
0808 18:31:06.495822 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 18:31:06.498034 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:06.479720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43957 (local address 127.29.78.130:60747)
0808 18:31:06.480368 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:06.480434 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:06.481715 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:06.482646 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:06.482657 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:06.483051 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:06.483841 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:06.483865 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:06.485130 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:06.485134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:06.486739 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:06.486752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:06.487640 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:06.487648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:06.487865 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:06.488585 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:06.488610 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:06.490654 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:06.493241 (+ 2587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:06.493251 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:06.493270 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:06.493697 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:06.494072 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:06.494076 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:06.494078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:06.494388 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:06.494987 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:06.494994 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:06.495336 (+ 342us) client_negotiation.cc:769] Sending connection context
0808 18:31:06.496504 (+ 1168us) client_negotiation.cc:240] Negotiation successful
0808 18:31:06.496860 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":158,"threads_started":1}
I20260808 18:31:06.501992 11481 heartbeater.cc:347] Connected to a master server at 127.29.78.190:43957
I20260808 18:31:06.502236 11481 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:06.502844 11481 heartbeater.cc:511] Master 127.29.78.190:43957 requested a full tablet report, sending...
I20260808 18:31:06.503973 11088 ts_manager.cc:194] Registered new tserver with Master: a4279ac13fc44ed2a1e345cc78656e7e (127.29.78.130:34905)
I20260808 18:31:06.504741 11088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:60747
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260808 18:31:06.535148 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:54763/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43957
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:40055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:31:06.682482 11487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:06.682955 11487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:06.683080 11487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:06.689189 11487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:06.689373 11487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:06.689535 11487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:31:06.697515 11487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:40055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.29.78.148:54763/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:06.699148 11487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:06.700578 11487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:06.712811 11495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:06.712911 11493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:06.713704 11492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:06.714377 11487 server_base.cc:1034] running on GCE node
I20260808 18:31:06.714885 11487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:06.715699 11487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:06.716858 11487 hybrid_clock.cc:648] HybridClock initialized: now 1786213866716788 us; error 89 us; skew 500 ppm
Aug 08 18:31:06 dist-test-slave-gp6n krb5kdc[10555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213866, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:06.720577 11487 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:31:06.721632 11487 webserver.cc:460] Webserver started at http://127.29.78.131:37333/ using document root <none> and password file <none>
I20260808 18:31:06.722333 11487 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:06.722436 11487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:06.722672 11487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 18:31:06 dist-test-slave-gp6n krb5kdc[10555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213866, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260808 18:31:06.740677 11487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "378f3a6febc3452bb08bb9110e664e4f"
format_stamp: "Formatted at 2026-08-08 18:31:06 on dist-test-slave-gp6n"
server_key: "fb83dc0fa9b15f6085b9c6018ebe6fd7"
server_key_iv: "28576cf38506c0c002a51e448916dca5"
server_key_version: "kuduclusterkey@0"
I20260808 18:31:06.742805 11487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "378f3a6febc3452bb08bb9110e664e4f"
format_stamp: "Formatted at 2026-08-08 18:31:06 on dist-test-slave-gp6n"
server_key: "fb83dc0fa9b15f6085b9c6018ebe6fd7"
server_key_iv: "28576cf38506c0c002a51e448916dca5"
server_key_version: "kuduclusterkey@0"
I20260808 18:31:06.748368 11487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 18:31:06.777110 11502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:06.779155 11487 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.003s sys 0.001s
I20260808 18:31:06.779302 11487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "378f3a6febc3452bb08bb9110e664e4f"
format_stamp: "Formatted at 2026-08-08 18:31:06 on dist-test-slave-gp6n"
server_key: "fb83dc0fa9b15f6085b9c6018ebe6fd7"
server_key_iv: "28576cf38506c0c002a51e448916dca5"
server_key_version: "kuduclusterkey@0"
I20260808 18:31:06.779456 11487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 08 18:31:06 dist-test-slave-gp6n krb5kdc[10555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213866, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260808 18:31:06.845593 11487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:06.850291 11487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:06.850536 11487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:06.851294 11487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:31:06.852349 11487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:31:06.852444 11487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:06.852510 11487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:31:06.852540 11487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:06.864498 11487 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:35095
I20260808 18:31:06.865659 11487 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:31:06.868180 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 11487
I20260808 18:31:06.868319 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:31:06.868651 30010 external_mini_cluster.cc:1428] Setting key d1a9f625839b754aaf93ec2ba49445fd
I20260808 18:31:06.870628 11633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:35095 every 8 connection(s)
Aug 08 18:31:06 dist-test-slave-gp6n krb5kdc[10555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213866, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:31:06.892719 11331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:06.872821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:54959 (local address 127.29.78.190:43957)
0808 18:31:06.872956 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:06.872961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:06.879559 (+ 6598us) server_negotiation.cc:407] Connection header received
0808 18:31:06.880662 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:06.880665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:06.880718 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:06.880810 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:06.882264 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:06.883122 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:06.884834 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:06.885186 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:06.885322 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:06.888495 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:06.888499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:06.888502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:06.888529 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:06.890290 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:06.890961 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:06.890965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:06.890967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:06.891056 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:06.891466 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:06.891469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:06.891471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:06.891675 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:06.891764 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:06.892442 (+ 678us) server_negotiation.cc:299] Negotiation successful
0808 18:31:06.892560 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 18:31:06.894078 11636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:06.873399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43957 (local address 127.29.78.131:54959)
0808 18:31:06.879354 (+ 5955us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:06.879415 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:06.880421 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:06.881017 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:06.881026 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:06.881375 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:06.882092 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:06.882111 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:06.883301 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:06.883305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:06.884701 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:06.884712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:06.885403 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:06.885409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:06.885665 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:06.886462 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:06.886482 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:06.888292 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:06.890441 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:06.890448 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:06.890465 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:06.890829 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:06.891157 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:06.891160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:06.891163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:06.891358 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:06.891795 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:06.891802 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:06.892151 (+ 349us) client_negotiation.cc:769] Sending connection context
0808 18:31:06.892371 (+ 220us) client_negotiation.cc:240] Negotiation successful
0808 18:31:06.892712 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5703,"thread_start_us":168,"threads_started":1}
I20260808 18:31:06.895533 11634 heartbeater.cc:347] Connected to a master server at 127.29.78.190:43957
I20260808 18:31:06.895799 11634 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:06.896360 11634 heartbeater.cc:511] Master 127.29.78.190:43957 requested a full tablet report, sending...
I20260808 18:31:06.897511 11088 ts_manager.cc:194] Registered new tserver with Master: 378f3a6febc3452bb08bb9110e664e4f (127.29.78.131:35095)
I20260808 18:31:06.898273 11088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:54959
I20260808 18:31:06.907353 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:31:06.908233 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 11183
I20260808 18:31:06.925256 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 11335
I20260808 18:31:06.931910 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 11487
I20260808 18:31:06.939494 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 11056
I20260808 18:31:06.948484 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:43957
--webserver_interface=127.29.78.190
--webserver_port=38445
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:40055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:43957
--ranger_config_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:54763/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.78.148:54763/kms with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:31:07.060307 11637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:07.060585 11637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:07.060627 11637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:07.064741 11637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:31:07.064803 11637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:07.064822 11637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:07.064839 11637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:31:07.064854 11637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:31:07.070197 11637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:40055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.78.148:54763/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:43957
--ranger_config_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:43957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=38445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:07.071242 11637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:07.072253 11637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:07.081040 11642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:07.081099 11643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:07.081091 11637 server_base.cc:1034] running on GCE node
W20260808 18:31:07.081115 11645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:07.081890 11637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:07.083002 11637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:07.084164 11637 hybrid_clock.cc:648] HybridClock initialized: now 1786213867084132 us; error 48 us; skew 500 ppm
Aug 08 18:31:07 dist-test-slave-gp6n krb5kdc[10555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213867, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:07.087134 11637 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:31:07.088050 11637 webserver.cc:460] Webserver started at http://127.29.78.190:38445/ using document root <none> and password file <none>
I20260808 18:31:07.088578 11637 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:07.088670 11637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 18:31:07 dist-test-slave-gp6n krb5kdc[10555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213867, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260808 18:31:07.139377 11637 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:31:07.141927 11653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:07.143404 11637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:31:07.143575 11637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "c6fb0b1f5e5c4e36bd62e3e657dc4ee6"
format_stamp: "Formatted at 2026-08-08 18:31:04 on dist-test-slave-gp6n"
server_key: "be35fb7dd3b158fc4b09c481b2a07f3e"
server_key_iv: "e4007e12861dd1b824bcb2e4588d2832"
server_key_version: "kuduclusterkey@0"
I20260808 18:31:07.144142 11637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:07.158255 11637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:07.162569 11637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:07.162824 11637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:07.173864 11710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:43957 every 8 connection(s)
I20260808 18:31:07.173883 11637 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:43957
I20260808 18:31:07.175269 11637 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:31:07.176510 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 11637
I20260808 18:31:07.177007 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:54763/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:41487
--local_ip_for_outbound_sockets=127.29.78.129
--tserver_master_addrs=127.29.78.190:43957
--webserver_port=39271
--webserver_interface=127.29.78.129
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:40055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.78.148:54763/kms with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:31:07.180289 11711 sys_catalog.cc:263] Verifying existing consensus state
I20260808 18:31:07.183003 11711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6: Bootstrap starting.
I20260808 18:31:07.206602 11711 log.cc:824] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:07.215036 11711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:31:07.215533 11711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6: Bootstrap complete.
I20260808 18:31:07.221130 11711 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6fb0b1f5e5c4e36bd62e3e657dc4ee6" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43957 } }
I20260808 18:31:07.221644 11711 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6fb0b1f5e5c4e36bd62e3e657dc4ee6, State: Initialized, Role: FOLLOWER
I20260808 18:31:07.222194 11711 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6fb0b1f5e5c4e36bd62e3e657dc4ee6" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43957 } }
I20260808 18:31:07.222357 11711 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:31:07.222457 11711 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:31:07.222576 11711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260808 18:31:07.225118 11711 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6fb0b1f5e5c4e36bd62e3e657dc4ee6" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43957 } }
I20260808 18:31:07.225482 11711 leader_election.cc:302] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6fb0b1f5e5c4e36bd62e3e657dc4ee6; no voters:
I20260808 18:31:07.225876 11711 leader_election.cc:288] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 18:31:07.226035 11715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 18:31:07.226315 11715 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [term 2 LEADER]: Becoming Leader. State: Replica: c6fb0b1f5e5c4e36bd62e3e657dc4ee6, State: Running, Role: LEADER
I20260808 18:31:07.226643 11715 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6fb0b1f5e5c4e36bd62e3e657dc4ee6" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43957 } }
I20260808 18:31:07.227193 11711 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:31:07.227983 11716 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c6fb0b1f5e5c4e36bd62e3e657dc4ee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6fb0b1f5e5c4e36bd62e3e657dc4ee6" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43957 } } }
I20260808 18:31:07.228106 11716 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [sys.catalog]: This master's current role is: LEADER
I20260808 18:31:07.228780 11717 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6fb0b1f5e5c4e36bd62e3e657dc4ee6. Latest consensus state: current_term: 2 leader_uuid: "c6fb0b1f5e5c4e36bd62e3e657dc4ee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6fb0b1f5e5c4e36bd62e3e657dc4ee6" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43957 } } }
I20260808 18:31:07.228874 11717 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6 [sys.catalog]: This master's current role is: LEADER
I20260808 18:31:07.228860 11726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:31:07.229548 11711 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 18:31:07.235817 11726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:31:07.236703 11726 catalog_manager.cc:1259] Loaded cluster ID: 1cec2c3ced0d43a7828504823d1d83a2
I20260808 18:31:07.236807 11726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:31:07.239652 11726 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:31:07.240406 11726 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6: Loaded TSK: 0
I20260808 18:31:07.241906 11726 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 18:31:07.335708 11713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:07.336047 11713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:07.336122 11713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:07.342569 11713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:07.342670 11713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:07.342772 11713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:31:07.351596 11713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:40055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.78.148:54763/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:41487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=39271
--tserver_master_addrs=127.29.78.190:43957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:07.353104 11713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:07.354575 11713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:07.368403 11753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:07.368700 11752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:07.369904 11713 server_base.cc:1034] running on GCE node
W20260808 18:31:07.369832 11755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:07.370571 11713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:07.371398 11713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:07.372604 11713 hybrid_clock.cc:648] HybridClock initialized: now 1786213867372529 us; error 88 us; skew 500 ppm
Aug 08 18:31:07 dist-test-slave-gp6n krb5kdc[10555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213867, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:07.376520 11713 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:31:07.377746 11713 webserver.cc:460] Webserver started at http://127.29.78.129:39271/ using document root <none> and password file <none>
I20260808 18:31:07.378515 11713 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:07.378628 11713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 18:31:07 dist-test-slave-gp6n krb5kdc[10555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213867, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260808 18:31:07.402878 11713 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.004s sys 0.000s
I20260808 18:31:07.406653 11763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:07.408358 11713 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 18:31:07.408502 11713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "d57e1e0961384085b45cc5d104b2b381"
format_stamp: "Formatted at 2026-08-08 18:31:05 on dist-test-slave-gp6n"
server_key: "c98edee75ef7e8d3659b9ab7b7041517"
server_key_iv: "14548cbbaf07183c3db4d3a86619666e"
server_key_version: "kuduclusterkey@0"
I20260808 18:31:07.409346 11713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:07.442008 11713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:07.445984 11713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:07.446200 11713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:07.447039 11713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:31:07.448135 11713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:31:07.448199 11713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:07.448259 11713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:31:07.448287 11713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:07.460978 11713 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:41487
I20260808 18:31:07.462384 11713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:31:07.467451 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 11713
I20260808 18:31:07.468041 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:54763/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:34905
--local_ip_for_outbound_sockets=127.29.78.130
--tserver_master_addrs=127.29.78.190:43957
--webserver_port=32979
--webserver_interface=127.29.78.130
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:40055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.78.148:54763/kms with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:31:07.477774 11891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:41487 every 8 connection(s)
Aug 08 18:31:07 dist-test-slave-gp6n krb5kdc[10555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213867, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:31:07.494894 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:07.472803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43957 (local address 127.29.78.129:53027)
0808 18:31:07.473353 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:07.473400 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:07.474464 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:07.476471 (+ 2007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:07.476481 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:07.476901 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:07.477474 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:07.477490 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:07.479678 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:07.479683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:07.481252 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:07.481265 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:07.482520 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:07.482525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:07.482744 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:07.483501 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:07.483528 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:07.487166 (+ 3638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:07.491151 (+ 3985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:07.491159 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:07.491175 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:07.491543 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:07.491983 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:07.491986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:07.491988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:07.492205 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:07.492978 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:07.492985 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:07.493288 (+ 303us) client_negotiation.cc:769] Sending connection context
0808 18:31:07.493469 (+ 181us) client_negotiation.cc:240] Negotiation successful
0808 18:31:07.493805 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":148,"threads_started":1}
I20260808 18:31:07.497479 11895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:07.472571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:53027 (local address 127.29.78.190:43957)
0808 18:31:07.473100 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:07.473127 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:07.473544 (+ 417us) server_negotiation.cc:407] Connection header received
0808 18:31:07.475084 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:07.475103 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:07.475556 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:07.475940 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:07.477666 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:07.479495 (+ 1829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:07.481457 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:07.482261 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:07.483206 (+ 945us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:07.487416 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:07.487424 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:07.487444 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:07.487480 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:07.491000 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:07.491699 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:07.491707 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:07.491713 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:07.491858 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:07.492332 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:07.492335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:07.492337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:07.492826 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:07.496089 (+ 3263us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:07.496245 (+ 156us) server_negotiation.cc:299] Negotiation successful
0808 18:31:07.496545 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":146,"threads_started":1}
I20260808 18:31:07.500566 11892 heartbeater.cc:347] Connected to a master server at 127.29.78.190:43957
I20260808 18:31:07.500829 11892 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:07.501413 11892 heartbeater.cc:511] Master 127.29.78.190:43957 requested a full tablet report, sending...
W20260808 18:31:07.503859 11892 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:43957 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 18:31:07.503995 11892 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:07.504163 11892 heartbeater.cc:511] Master 127.29.78.190:43957 requested a full tablet report, sending...
I20260808 18:31:07.505149 11892 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:07.505316 11892 heartbeater.cc:511] Master 127.29.78.190:43957 requested a full tablet report, sending...
W20260808 18:31:07.506131 11892 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 18:31:07.618170 11894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:07.618486 11894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:07.618567 11894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:07.622586 11894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:07.622637 11894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:07.622692 11894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:31:07.628604 11894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:40055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.78.148:54763/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:34905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=32979
--tserver_master_addrs=127.29.78.190:43957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:07.629675 11894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:07.631031 11894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:07.643700 11903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:07.643751 11902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:07.644340 11905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:07.645479 11894 server_base.cc:1034] running on GCE node
I20260808 18:31:07.645990 11894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:07.646831 11894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:07.648116 11894 hybrid_clock.cc:648] HybridClock initialized: now 1786213867648047 us; error 87 us; skew 500 ppm
Aug 08 18:31:07 dist-test-slave-gp6n krb5kdc[10555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213867, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:07.652077 11894 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:31:07.653281 11894 webserver.cc:460] Webserver started at http://127.29.78.130:32979/ using document root <none> and password file <none>
I20260808 18:31:07.654098 11894 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:07.654219 11894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 18:31:07 dist-test-slave-gp6n krb5kdc[10555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213867, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260808 18:31:07.684715 11894 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260808 18:31:07.691265 11916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:07.693027 11894 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260808 18:31:07.693166 11894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "a4279ac13fc44ed2a1e345cc78656e7e"
format_stamp: "Formatted at 2026-08-08 18:31:06 on dist-test-slave-gp6n"
server_key: "cbce267690434a04a4255d5688d43fde"
server_key_iv: "eb83412e40877433de899e337c64cdb4"
server_key_version: "kuduclusterkey@0"
I20260808 18:31:07.693976 11894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:07.721889 11894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:07.725808 11894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:07.726018 11894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:07.726763 11894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:31:07.727903 11894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:31:07.727986 11894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:07.728047 11894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:31:07.728155 11894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260808 18:31:07.740967 11894 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:34905
I20260808 18:31:07.741398 12042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:34905 every 8 connection(s)
I20260808 18:31:07.742339 11894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:31:07.746019 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 11894
I20260808 18:31:07.746600 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:54763/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:35095
--local_ip_for_outbound_sockets=127.29.78.131
--tserver_master_addrs=127.29.78.190:43957
--webserver_port=37333
--webserver_interface=127.29.78.131
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:40055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.78.148:54763/kms with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 18:31:07 dist-test-slave-gp6n krb5kdc[10555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213867, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:31:07.764740 11895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:07.744542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:42745 (local address 127.29.78.190:43957)
0808 18:31:07.746060 (+ 1518us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:07.746065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:07.746084 (+ 19us) server_negotiation.cc:407] Connection header received
0808 18:31:07.752412 (+ 6328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:07.752416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:07.752481 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:07.752579 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:07.754275 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:07.755140 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:07.756883 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:07.757317 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:07.757479 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:07.760419 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:07.760423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:07.760426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:07.760454 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:07.762261 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:07.762938 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:07.762942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:07.762944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:07.763036 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:07.763474 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:07.763478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:07.763479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:07.763699 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:07.763795 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:07.764441 (+ 646us) server_negotiation.cc:299] Negotiation successful
0808 18:31:07.764570 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1421}
I20260808 18:31:07.766075 12045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:07.744977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43957 (local address 127.29.78.130:42745)
0808 18:31:07.745516 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:07.745560 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:07.746743 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:07.752917 (+ 6174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:07.752939 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:07.753323 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:07.754070 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:07.754096 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:07.755300 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:07.755303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:07.756740 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:07.756751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:07.757537 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:07.757543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:07.757753 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:07.758556 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:07.758572 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:07.760239 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:07.762436 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:07.762443 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:07.762458 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:07.762825 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:07.763142 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:07.763145 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:07.763147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:07.763347 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:07.763827 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:07.763836 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:07.764155 (+ 319us) client_negotiation.cc:769] Sending connection context
0808 18:31:07.764365 (+ 210us) client_negotiation.cc:240] Negotiation successful
0808 18:31:07.764718 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":151,"threads_started":1}
I20260808 18:31:07.767459 12043 heartbeater.cc:347] Connected to a master server at 127.29.78.190:43957
I20260808 18:31:07.767731 12043 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:07.768678 12043 heartbeater.cc:511] Master 127.29.78.190:43957 requested a full tablet report, sending...
W20260808 18:31:07.770373 12043 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:43957 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 18:31:07.770524 12043 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:07.770674 12043 heartbeater.cc:511] Master 127.29.78.190:43957 requested a full tablet report, sending...
I20260808 18:31:07.771474 12043 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:07.771627 12043 heartbeater.cc:511] Master 127.29.78.190:43957 requested a full tablet report, sending...
W20260808 18:31:07.772353 12043 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 18:31:07.888798 12046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:07.889165 12046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:07.889237 12046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:07.895291 12046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:07.895380 12046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:07.895470 12046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:31:07.903268 12046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:40055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.78.148:54763/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:35095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=37333
--tserver_master_addrs=127.29.78.190:43957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:07.904567 12046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:07.905988 12046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:07.915771 12054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:07.915757 12051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:07.916069 12052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:07.916524 12046 server_base.cc:1034] running on GCE node
I20260808 18:31:07.916935 12046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:07.917618 12046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:07.918797 12046 hybrid_clock.cc:648] HybridClock initialized: now 1786213867918770 us; error 41 us; skew 500 ppm
Aug 08 18:31:07 dist-test-slave-gp6n krb5kdc[10555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213867, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:07.922207 12046 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:31:07.923384 12046 webserver.cc:460] Webserver started at http://127.29.78.131:37333/ using document root <none> and password file <none>
I20260808 18:31:07.923964 12046 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:07.924070 12046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 18:31:07 dist-test-slave-gp6n krb5kdc[10555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213867, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260808 18:31:07.947990 12046 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260808 18:31:07.954394 12062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:07.956303 12046 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260808 18:31:07.956522 12046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "378f3a6febc3452bb08bb9110e664e4f"
format_stamp: "Formatted at 2026-08-08 18:31:06 on dist-test-slave-gp6n"
server_key: "fb83dc0fa9b15f6085b9c6018ebe6fd7"
server_key_iv: "28576cf38506c0c002a51e448916dca5"
server_key_version: "kuduclusterkey@0"
I20260808 18:31:07.957356 12046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:07.989084 12046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:07.992537 12046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:07.992731 12046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:07.993400 12046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:31:07.994493 12046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:31:07.994558 12046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:07.994616 12046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:31:07.994648 12046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:08.006732 12046 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:35095
I20260808 18:31:08.006803 12199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:35095 every 8 connection(s)
I20260808 18:31:08.008172 12046 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:31:08.016360 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 12046
Aug 08 18:31:08 dist-test-slave-gp6n krb5kdc[10555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213867, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:31:08.024753 11895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:08.010113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:60737 (local address 127.29.78.190:43957)
0808 18:31:08.010281 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:08.010286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:08.011229 (+ 943us) server_negotiation.cc:407] Connection header received
0808 18:31:08.012177 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:08.012181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:08.012238 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:08.012360 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:08.013461 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.014306 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.015683 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.016040 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.016159 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:08.020018 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:08.020029 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:08.020032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:08.020056 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:08.021725 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:08.022390 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:08.022447 (+ 57us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:08.022449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:08.022535 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:08.022917 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:08.022920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:08.022922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:08.023098 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:08.023210 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:08.024096 (+ 886us) server_negotiation.cc:299] Negotiation successful
0808 18:31:08.024281 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 18:31:08.025306 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:08.010562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43957 (local address 127.29.78.131:60737)
0808 18:31:08.011052 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:08.011105 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:08.011958 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:08.012510 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:08.012517 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:08.012786 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:08.013266 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:08.013279 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.014517 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.014521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:08.015561 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:08.015570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.016213 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.016217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:08.016424 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:08.017721 (+ 1297us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:08.017742 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:08.019847 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:08.021847 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:08.021854 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:08.021872 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:08.022269 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:08.022639 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:08.022642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:08.022644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:08.022809 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:08.023209 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:08.023215 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:08.023523 (+ 308us) client_negotiation.cc:769] Sending connection context
0808 18:31:08.023744 (+ 221us) client_negotiation.cc:240] Negotiation successful
0808 18:31:08.024130 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":135,"threads_started":1}
I20260808 18:31:08.026593 12200 heartbeater.cc:347] Connected to a master server at 127.29.78.190:43957
I20260808 18:31:08.026865 12200 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:08.027477 12200 heartbeater.cc:511] Master 127.29.78.190:43957 requested a full tablet report, sending...
I20260808 18:31:08.027524 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:08.017403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.29.78.190:43957)
0808 18:31:08.017689 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:08.017697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:08.017711 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:31:08.017749 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:08.017752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:08.017796 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:08.017902 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:08.018969 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.019515 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.020767 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.021104 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.021228 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:08.023538 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:08.023542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:08.023544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:08.023566 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:08.025236 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:08.025758 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:08.025761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:08.025763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:08.025856 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:08.026495 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:08.026498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:08.026500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:08.026725 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:08.026949 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:08.027225 (+ 276us) server_negotiation.cc:299] Negotiation successful
0808 18:31:08.027367 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":137,"threads_started":1}
I20260808 18:31:08.029036 11672 ts_manager.cc:194] Registered new tserver with Master: 378f3a6febc3452bb08bb9110e664e4f (127.29.78.131:35095)
I20260808 18:31:08.030275 11672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:60737
Aug 08 18:31:08 dist-test-slave-gp6n krb5kdc[10555](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786213868, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:08.506387 11892 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:08.506721 11892 heartbeater.cc:511] Master 127.29.78.190:43957 requested a full tablet report, sending...
I20260808 18:31:08.507464 11672 ts_manager.cc:194] Registered new tserver with Master: d57e1e0961384085b45cc5d104b2b381 (127.29.78.129:41487)
I20260808 18:31:08.508312 11672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:53027
Aug 08 18:31:08 dist-test-slave-gp6n krb5kdc[10555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213868, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260808 18:31:08.772583 12043 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:08.772903 12043 heartbeater.cc:511] Master 127.29.78.190:43957 requested a full tablet report, sending...
I20260808 18:31:08.773645 11672 ts_manager.cc:194] Registered new tserver with Master: a4279ac13fc44ed2a1e345cc78656e7e (127.29.78.130:34905)
I20260808 18:31:08.774501 11672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:42745
I20260808 18:31:08.779807 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:31:08.791810 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:08.781516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41604 (local address 127.29.78.190:43957)
0808 18:31:08.782104 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:08.782108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:08.782435 (+ 327us) server_negotiation.cc:407] Connection header received
0808 18:31:08.782620 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:08.782623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:08.782688 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:08.782747 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:08.783730 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.784555 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.785744 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.786102 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.786172 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:08.788057 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:08.788062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:08.788065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:08.788090 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:08.789896 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:08.790392 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:08.790396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:08.790420 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:08.790517 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:08.790829 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:08.790833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:08.790835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:08.791059 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:08.791209 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:08.791497 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 18:31:08.791629 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":100,"threads_started":1}
I20260808 18:31:08.795528 11672 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:31:08.799054 11672 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:31:08.818782 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:08.809588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:34905 (local address 127.0.0.1:57392)
0808 18:31:08.810494 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:08.810544 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:08.810672 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:08.811603 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:08.811609 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:08.811634 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:08.811785 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:08.811792 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.813026 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.813030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:08.814219 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:08.814228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.818218 (+ 3990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.818221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:08.818334 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:08.818380 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 18:31:08.818497 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 18:31:08.818616 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":109,"threads_started":1}
I20260808 18:31:08.819653 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:08.810807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57392 (local address 127.29.78.130:34905)
0808 18:31:08.811086 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:08.811096 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:08.811122 (+ 26us) server_negotiation.cc:407] Connection header received
0808 18:31:08.811198 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:08.811204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:08.811429 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:08.811958 (+ 529us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:08.812130 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.812907 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.814355 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.815205 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.819180 (+ 3975us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:08.819339 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:08.819390 (+ 51us) server_negotiation.cc:299] Negotiation successful
0808 18:31:08.819486 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":103,"threads_started":1}
I20260808 18:31:08.821684 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:08.810048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:41487 (local address 127.0.0.1:44870)
0808 18:31:08.811796 (+ 1748us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:08.811809 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:08.811902 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:08.815969 (+ 4067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:08.815972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:08.815989 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:08.816074 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:08.816079 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.818192 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.818197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:08.819544 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:08.819554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.821272 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.821277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:08.821392 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:08.821417 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 18:31:08.821488 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 18:31:08.821555 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1687,"thread_start_us":100,"threads_started":1}
I20260808 18:31:08.822266 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:08.810495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44870 (local address 127.29.78.129:41487)
0808 18:31:08.815378 (+ 4883us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:08.815385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:08.815408 (+ 23us) server_negotiation.cc:407] Connection header received
0808 18:31:08.815487 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:08.815495 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:08.815688 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:08.815826 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:08.816198 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.818060 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.819687 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.820958 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.821061 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:08.821208 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:08.822022 (+ 814us) server_negotiation.cc:299] Negotiation successful
0808 18:31:08.822122 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4757,"thread_start_us":94,"threads_started":1}
I20260808 18:31:08.822687 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:08.810355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:35095 (local address 127.0.0.1:36488)
0808 18:31:08.812071 (+ 1716us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:08.812084 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:08.812161 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:08.815318 (+ 3157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:08.815321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:08.815338 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:08.815448 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:08.815453 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.816862 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.816875 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:08.818059 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:08.818069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.822242 (+ 4173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.822246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:08.822353 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:08.822382 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 18:31:08.822471 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 18:31:08.822554 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1662,"thread_start_us":751,"threads_started":1}
I20260808 18:31:08.823232 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:08.810843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36488 (local address 127.29.78.131:35095)
0808 18:31:08.814633 (+ 3790us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:08.814640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:08.814663 (+ 23us) server_negotiation.cc:407] Connection header received
0808 18:31:08.814734 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:08.814739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:08.814926 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:08.815071 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:08.815585 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.816727 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.820561 (+ 3834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.821950 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.822059 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:08.822180 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:08.823042 (+ 862us) server_negotiation.cc:299] Negotiation successful
0808 18:31:08.823123 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3682,"thread_start_us":96,"threads_started":1}
I20260808 18:31:08.822927 11969 tablet_service.cc:1467] Processing CreateTablet for tablet 26ac63fa60d64d50b8f4d7a7410954e7 (DEFAULT_TABLE table=test-table [id=1e34446ec36c4f278b047ba4712c488b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:31:08.824150 11969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ac63fa60d64d50b8f4d7a7410954e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:31:08.828703 12107 tablet_service.cc:1467] Processing CreateTablet for tablet 26ac63fa60d64d50b8f4d7a7410954e7 (DEFAULT_TABLE table=test-table [id=1e34446ec36c4f278b047ba4712c488b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:31:08.830108 12107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ac63fa60d64d50b8f4d7a7410954e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:31:08.832144 11816 tablet_service.cc:1467] Processing CreateTablet for tablet 26ac63fa60d64d50b8f4d7a7410954e7 (DEFAULT_TABLE table=test-table [id=1e34446ec36c4f278b047ba4712c488b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:31:08.833213 11816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ac63fa60d64d50b8f4d7a7410954e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:31:08.844753 12230 tablet_bootstrap.cc:492] T 26ac63fa60d64d50b8f4d7a7410954e7 P d57e1e0961384085b45cc5d104b2b381: Bootstrap starting.
I20260808 18:31:08.848522 12230 tablet_bootstrap.cc:654] T 26ac63fa60d64d50b8f4d7a7410954e7 P d57e1e0961384085b45cc5d104b2b381: Neither blocks nor log segments found. Creating new log.
I20260808 18:31:08.849143 12229 tablet_bootstrap.cc:492] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f: Bootstrap starting.
I20260808 18:31:08.849638 12230 log.cc:824] T 26ac63fa60d64d50b8f4d7a7410954e7 P d57e1e0961384085b45cc5d104b2b381: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:08.850370 12231 tablet_bootstrap.cc:492] T 26ac63fa60d64d50b8f4d7a7410954e7 P a4279ac13fc44ed2a1e345cc78656e7e: Bootstrap starting.
I20260808 18:31:08.851598 12229 tablet_bootstrap.cc:654] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f: Neither blocks nor log segments found. Creating new log.
I20260808 18:31:08.852449 12229 log.cc:824] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:08.852792 12231 tablet_bootstrap.cc:654] T 26ac63fa60d64d50b8f4d7a7410954e7 P a4279ac13fc44ed2a1e345cc78656e7e: Neither blocks nor log segments found. Creating new log.
I20260808 18:31:08.853624 12231 log.cc:824] T 26ac63fa60d64d50b8f4d7a7410954e7 P a4279ac13fc44ed2a1e345cc78656e7e: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:08.856170 12231 tablet_bootstrap.cc:492] T 26ac63fa60d64d50b8f4d7a7410954e7 P a4279ac13fc44ed2a1e345cc78656e7e: No bootstrap required, opened a new log
I20260808 18:31:08.856423 12231 ts_tablet_manager.cc:1397] T 26ac63fa60d64d50b8f4d7a7410954e7 P a4279ac13fc44ed2a1e345cc78656e7e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:31:08.858179 12230 tablet_bootstrap.cc:492] T 26ac63fa60d64d50b8f4d7a7410954e7 P d57e1e0961384085b45cc5d104b2b381: No bootstrap required, opened a new log
I20260808 18:31:08.858420 12230 ts_tablet_manager.cc:1397] T 26ac63fa60d64d50b8f4d7a7410954e7 P d57e1e0961384085b45cc5d104b2b381: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260808 18:31:08.860944 12231 raft_consensus.cc:348] T 26ac63fa60d64d50b8f4d7a7410954e7 P a4279ac13fc44ed2a1e345cc78656e7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4279ac13fc44ed2a1e345cc78656e7e" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34905 } } peers { permanent_uuid: "378f3a6febc3452bb08bb9110e664e4f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35095 } } peers { permanent_uuid: "d57e1e0961384085b45cc5d104b2b381" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 41487 } }
I20260808 18:31:08.861438 12231 raft_consensus.cc:374] T 26ac63fa60d64d50b8f4d7a7410954e7 P a4279ac13fc44ed2a1e345cc78656e7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:31:08.861542 12231 raft_consensus.cc:729] T 26ac63fa60d64d50b8f4d7a7410954e7 P a4279ac13fc44ed2a1e345cc78656e7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4279ac13fc44ed2a1e345cc78656e7e, State: Initialized, Role: FOLLOWER
I20260808 18:31:08.862123 12231 consensus_queue.cc:260] T 26ac63fa60d64d50b8f4d7a7410954e7 P a4279ac13fc44ed2a1e345cc78656e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4279ac13fc44ed2a1e345cc78656e7e" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34905 } } peers { permanent_uuid: "378f3a6febc3452bb08bb9110e664e4f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35095 } } peers { permanent_uuid: "d57e1e0961384085b45cc5d104b2b381" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 41487 } }
I20260808 18:31:08.863569 12231 ts_tablet_manager.cc:1428] T 26ac63fa60d64d50b8f4d7a7410954e7 P a4279ac13fc44ed2a1e345cc78656e7e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260808 18:31:08.863751 12230 raft_consensus.cc:348] T 26ac63fa60d64d50b8f4d7a7410954e7 P d57e1e0961384085b45cc5d104b2b381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4279ac13fc44ed2a1e345cc78656e7e" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34905 } } peers { permanent_uuid: "378f3a6febc3452bb08bb9110e664e4f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35095 } } peers { permanent_uuid: "d57e1e0961384085b45cc5d104b2b381" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 41487 } }
I20260808 18:31:08.863974 12043 heartbeater.cc:503] Master 127.29.78.190:43957 was elected leader, sending a full tablet report...
I20260808 18:31:08.864009 12230 raft_consensus.cc:374] T 26ac63fa60d64d50b8f4d7a7410954e7 P d57e1e0961384085b45cc5d104b2b381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:31:08.864063 12230 raft_consensus.cc:729] T 26ac63fa60d64d50b8f4d7a7410954e7 P d57e1e0961384085b45cc5d104b2b381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d57e1e0961384085b45cc5d104b2b381, State: Initialized, Role: FOLLOWER
I20260808 18:31:08.864532 12230 consensus_queue.cc:260] T 26ac63fa60d64d50b8f4d7a7410954e7 P d57e1e0961384085b45cc5d104b2b381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4279ac13fc44ed2a1e345cc78656e7e" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34905 } } peers { permanent_uuid: "378f3a6febc3452bb08bb9110e664e4f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35095 } } peers { permanent_uuid: "d57e1e0961384085b45cc5d104b2b381" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 41487 } }
I20260808 18:31:08.865536 12230 ts_tablet_manager.cc:1428] T 26ac63fa60d64d50b8f4d7a7410954e7 P d57e1e0961384085b45cc5d104b2b381: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260808 18:31:08.865857 11892 heartbeater.cc:503] Master 127.29.78.190:43957 was elected leader, sending a full tablet report...
I20260808 18:31:08.871690 12229 tablet_bootstrap.cc:492] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f: No bootstrap required, opened a new log
I20260808 18:31:08.871904 12229 ts_tablet_manager.cc:1397] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20260808 18:31:08.876327 12229 raft_consensus.cc:348] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4279ac13fc44ed2a1e345cc78656e7e" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34905 } } peers { permanent_uuid: "378f3a6febc3452bb08bb9110e664e4f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35095 } } peers { permanent_uuid: "d57e1e0961384085b45cc5d104b2b381" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 41487 } }
I20260808 18:31:08.876590 12229 raft_consensus.cc:374] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:31:08.876646 12229 raft_consensus.cc:729] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 378f3a6febc3452bb08bb9110e664e4f, State: Initialized, Role: FOLLOWER
I20260808 18:31:08.877077 12229 consensus_queue.cc:260] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4279ac13fc44ed2a1e345cc78656e7e" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34905 } } peers { permanent_uuid: "378f3a6febc3452bb08bb9110e664e4f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35095 } } peers { permanent_uuid: "d57e1e0961384085b45cc5d104b2b381" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 41487 } }
I20260808 18:31:08.877764 12200 heartbeater.cc:503] Master 127.29.78.190:43957 was elected leader, sending a full tablet report...
I20260808 18:31:08.878268 12229 ts_tablet_manager.cc:1428] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260808 18:31:08.954511 12237 raft_consensus.cc:482] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:31:08.954756 12237 raft_consensus.cc:504] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4279ac13fc44ed2a1e345cc78656e7e" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34905 } } peers { permanent_uuid: "378f3a6febc3452bb08bb9110e664e4f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35095 } } peers { permanent_uuid: "d57e1e0961384085b45cc5d104b2b381" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 41487 } }
I20260808 18:31:08.956936 12237 leader_election.cc:288] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4279ac13fc44ed2a1e345cc78656e7e (127.29.78.130:34905), d57e1e0961384085b45cc5d104b2b381 (127.29.78.129:41487)
I20260808 18:31:08.961619 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:08.956856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:35769 (local address 127.29.78.129:41487)
0808 18:31:08.957038 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:08.957043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:08.957339 (+ 296us) server_negotiation.cc:407] Connection header received
0808 18:31:08.957538 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:08.957541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:08.957586 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:08.957629 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:08.958055 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.958850 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.960385 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.960963 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.961052 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:08.961103 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:08.961415 (+ 312us) server_negotiation.cc:299] Negotiation successful
0808 18:31:08.961494 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 18:31:08.962239 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:08.956434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:41487 (local address 127.29.78.131:35769)
0808 18:31:08.957243 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:08.957258 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:08.957422 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:08.957760 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:08.957763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:08.957791 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:08.957933 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:08.957939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.959002 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.959005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:08.960246 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:08.960256 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.961192 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.961195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:08.961288 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:08.961309 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 18:31:08.962020 (+ 711us) client_negotiation.cc:240] Negotiation successful
0808 18:31:08.962095 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":139,"threads_started":1}
I20260808 18:31:08.963047 11838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ac63fa60d64d50b8f4d7a7410954e7" candidate_uuid: "378f3a6febc3452bb08bb9110e664e4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d57e1e0961384085b45cc5d104b2b381" is_pre_election: true
I20260808 18:31:08.963351 11838 raft_consensus.cc:2454] T 26ac63fa60d64d50b8f4d7a7410954e7 P d57e1e0961384085b45cc5d104b2b381 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 378f3a6febc3452bb08bb9110e664e4f in term 0.
I20260808 18:31:08.964725 12065 leader_election.cc:302] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 378f3a6febc3452bb08bb9110e664e4f, d57e1e0961384085b45cc5d104b2b381; no voters:
I20260808 18:31:08.965261 12237 raft_consensus.cc:2781] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:31:08.965350 12237 raft_consensus.cc:482] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:31:08.965397 12237 raft_consensus.cc:3037] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:31:08.966760 12237 raft_consensus.cc:504] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4279ac13fc44ed2a1e345cc78656e7e" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34905 } } peers { permanent_uuid: "378f3a6febc3452bb08bb9110e664e4f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35095 } } peers { permanent_uuid: "d57e1e0961384085b45cc5d104b2b381" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 41487 } }
I20260808 18:31:08.967131 12237 leader_election.cc:288] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f [CANDIDATE]: Term 1 election: Requested vote from peers a4279ac13fc44ed2a1e345cc78656e7e (127.29.78.130:34905), d57e1e0961384085b45cc5d104b2b381 (127.29.78.129:41487)
I20260808 18:31:08.968171 11838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ac63fa60d64d50b8f4d7a7410954e7" candidate_uuid: "378f3a6febc3452bb08bb9110e664e4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d57e1e0961384085b45cc5d104b2b381"
I20260808 18:31:08.968302 11838 raft_consensus.cc:3037] T 26ac63fa60d64d50b8f4d7a7410954e7 P d57e1e0961384085b45cc5d104b2b381 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:31:08.969619 11838 raft_consensus.cc:2454] T 26ac63fa60d64d50b8f4d7a7410954e7 P d57e1e0961384085b45cc5d104b2b381 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 378f3a6febc3452bb08bb9110e664e4f in term 1.
I20260808 18:31:08.969993 12065 leader_election.cc:302] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 378f3a6febc3452bb08bb9110e664e4f, d57e1e0961384085b45cc5d104b2b381; no voters:
I20260808 18:31:08.970216 12237 raft_consensus.cc:2781] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:31:08.970219 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:08.956954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:53605 (local address 127.29.78.130:34905)
0808 18:31:08.957139 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:08.957143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:08.959825 (+ 2682us) server_negotiation.cc:407] Connection header received
0808 18:31:08.963158 (+ 3333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:08.963161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:08.963214 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:08.963263 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:08.963649 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.964409 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.968810 (+ 4401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.969455 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.969558 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:08.969611 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:08.969984 (+ 373us) server_negotiation.cc:299] Negotiation successful
0808 18:31:08.970090 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260808 18:31:08.970446 12237 raft_consensus.cc:686] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f [term 1 LEADER]: Becoming Leader. State: Replica: 378f3a6febc3452bb08bb9110e664e4f, State: Running, Role: LEADER
I20260808 18:31:08.970701 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:08.956440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:34905 (local address 127.29.78.131:53605)
0808 18:31:08.959724 (+ 3284us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:08.959740 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:08.962996 (+ 3256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:08.963401 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:08.963404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:08.963421 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:08.963520 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:08.963526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.964585 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.964588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:08.965743 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:08.965755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:08.969715 (+ 3960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:08.969719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:08.969825 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:08.969847 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 18:31:08.970468 (+ 621us) client_negotiation.cc:240] Negotiation successful
0808 18:31:08.970545 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3201,"mutex_wait_us":16,"thread_start_us":135,"threads_started":1}
I20260808 18:31:08.970789 12237 consensus_queue.cc:237] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4279ac13fc44ed2a1e345cc78656e7e" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34905 } } peers { permanent_uuid: "378f3a6febc3452bb08bb9110e664e4f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35095 } } peers { permanent_uuid: "d57e1e0961384085b45cc5d104b2b381" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 41487 } }
I20260808 18:31:08.971536 11988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ac63fa60d64d50b8f4d7a7410954e7" candidate_uuid: "378f3a6febc3452bb08bb9110e664e4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4279ac13fc44ed2a1e345cc78656e7e"
I20260808 18:31:08.971540 11990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ac63fa60d64d50b8f4d7a7410954e7" candidate_uuid: "378f3a6febc3452bb08bb9110e664e4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4279ac13fc44ed2a1e345cc78656e7e" is_pre_election: true
I20260808 18:31:08.971799 11988 raft_consensus.cc:3037] T 26ac63fa60d64d50b8f4d7a7410954e7 P a4279ac13fc44ed2a1e345cc78656e7e [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:31:08.973127 11988 raft_consensus.cc:2454] T 26ac63fa60d64d50b8f4d7a7410954e7 P a4279ac13fc44ed2a1e345cc78656e7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 378f3a6febc3452bb08bb9110e664e4f in term 1.
W20260808 18:31:08.973459 11893 tablet.cc:2369] T 26ac63fa60d64d50b8f4d7a7410954e7 P d57e1e0961384085b45cc5d104b2b381: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:31:08.975843 11672 catalog_manager.cc:5495] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 378f3a6febc3452bb08bb9110e664e4f (127.29.78.131). New cstate: current_term: 1 leader_uuid: "378f3a6febc3452bb08bb9110e664e4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4279ac13fc44ed2a1e345cc78656e7e" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "378f3a6febc3452bb08bb9110e664e4f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d57e1e0961384085b45cc5d104b2b381" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 41487 } health_report { overall_health: UNKNOWN } } }
W20260808 18:31:08.995113 12044 tablet.cc:2369] T 26ac63fa60d64d50b8f4d7a7410954e7 P a4279ac13fc44ed2a1e345cc78656e7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 18:31:08 dist-test-slave-gp6n krb5kdc[10555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213868, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
W20260808 18:31:09.009963 12201 tablet.cc:2369] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:31:09.022991 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:09.018770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36498 (local address 127.29.78.131:35095)
0808 18:31:09.018889 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:09.018894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:09.018971 (+ 77us) server_negotiation.cc:407] Connection header received
0808 18:31:09.019128 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:09.019132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:09.019207 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:09.019333 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:31:09.019744 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:09.020537 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:09.021480 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:09.021709 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:09.021833 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:09.022142 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:31:09.022594 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:31:09.022724 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:09.022787 (+ 63us) server_negotiation.cc:299] Negotiation successful
0808 18:31:09.022852 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 18:31:09.030172 11838 raft_consensus.cc:1261] T 26ac63fa60d64d50b8f4d7a7410954e7 P d57e1e0961384085b45cc5d104b2b381 [term 1 FOLLOWER]: Refusing update from remote peer 378f3a6febc3452bb08bb9110e664e4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:31:09.030844 11988 raft_consensus.cc:1261] T 26ac63fa60d64d50b8f4d7a7410954e7 P a4279ac13fc44ed2a1e345cc78656e7e [term 1 FOLLOWER]: Refusing update from remote peer 378f3a6febc3452bb08bb9110e664e4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:31:09.031486 12240 consensus_queue.cc:1035] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4279ac13fc44ed2a1e345cc78656e7e" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:31:09.031631 12237 consensus_queue.cc:1035] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d57e1e0961384085b45cc5d104b2b381" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 41487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:31:09.043066 12248 mvcc.cc:205] Tried to move back new op lower bound from 7316332007539298304 to 7316332007315148800. Current Snapshot: MvccSnapshot[applied={T|T < 7316332007539298304}]
I20260808 18:31:09.057282 11668 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:31:09.057494 11668 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:31:09.059681 12247 mvcc.cc:205] Tried to move back new op lower bound from 7316332007539298304 to 7316332007315148800. Current Snapshot: MvccSnapshot[applied={T|T < 7316332007539298304 or (T in {7316332007539298304})}]
I20260808 18:31:09.060535 11668 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6fb0b1f5e5c4e36bd62e3e657dc4ee6: Sending DeleteTablet for 3 replicas of tablet 26ac63fa60d64d50b8f4d7a7410954e7
I20260808 18:31:09.061383 11969 tablet_service.cc:1514] Processing DeleteTablet for tablet 26ac63fa60d64d50b8f4d7a7410954e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:31:09 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:57392
I20260808 18:31:09.062377 12107 tablet_service.cc:1514] Processing DeleteTablet for tablet 26ac63fa60d64d50b8f4d7a7410954e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:31:09 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:36488
I20260808 18:31:09.063395 11816 tablet_service.cc:1514] Processing DeleteTablet for tablet 26ac63fa60d64d50b8f4d7a7410954e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:31:09 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:44870
I20260808 18:31:09.065716 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 11713
I20260808 18:31:09.090677 12257 tablet_replica.cc:330] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f: stopping tablet replica
I20260808 18:31:09.091665 12257 raft_consensus.cc:2229] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:31:09.092213 12257 raft_consensus.cc:2258] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:31:09.092571 12256 tablet_replica.cc:330] T 26ac63fa60d64d50b8f4d7a7410954e7 P a4279ac13fc44ed2a1e345cc78656e7e: stopping tablet replica
I20260808 18:31:09.092963 12256 raft_consensus.cc:2229] T 26ac63fa60d64d50b8f4d7a7410954e7 P a4279ac13fc44ed2a1e345cc78656e7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:31:09.093613 12256 raft_consensus.cc:2258] T 26ac63fa60d64d50b8f4d7a7410954e7 P a4279ac13fc44ed2a1e345cc78656e7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:31:09.095623 12257 ts_tablet_manager.cc:1905] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:31:09.099007 12257 ts_tablet_manager.cc:1918] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:31:09.100533 12256 ts_tablet_manager.cc:1905] T 26ac63fa60d64d50b8f4d7a7410954e7 P a4279ac13fc44ed2a1e345cc78656e7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:31:09.103317 12256 ts_tablet_manager.cc:1918] T 26ac63fa60d64d50b8f4d7a7410954e7 P a4279ac13fc44ed2a1e345cc78656e7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260808 18:31:09.106297 11655 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:41487 (error 108)
I20260808 18:31:09.107324 12256 log.cc:1196] T 26ac63fa60d64d50b8f4d7a7410954e7 P a4279ac13fc44ed2a1e345cc78656e7e: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/26ac63fa60d64d50b8f4d7a7410954e7
I20260808 18:31:09.107630 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 11894
I20260808 18:31:09.115437 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:09.115069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:41487 (local address 127.0.0.1:44882)
0808 18:31:09.115264 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:09.115340 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:41487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260808 18:31:09.116202 11655 catalog_manager.cc:4584] TS d57e1e0961384085b45cc5d104b2b381 (127.29.78.129:41487): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 26ac63fa60d64d50b8f4d7a7410954e7: Network error: Client connection negotiation failed: client connection to 127.29.78.129:41487: connect: Connection refused (error 111)
I20260808 18:31:09.117049 12257 log.cc:1196] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/26ac63fa60d64d50b8f4d7a7410954e7
I20260808 18:31:09.117753 12257 ts_tablet_manager.cc:1939] T 26ac63fa60d64d50b8f4d7a7410954e7 P 378f3a6febc3452bb08bb9110e664e4f: Deleting consensus metadata
I20260808 18:31:09.120137 11658 catalog_manager.cc:4841] TS 378f3a6febc3452bb08bb9110e664e4f (127.29.78.131:35095): tablet 26ac63fa60d64d50b8f4d7a7410954e7 (table test-table [id=1e34446ec36c4f278b047ba4712c488b]) successfully deleted
I20260808 18:31:09.128254 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 12046
I20260808 18:31:09.128497 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:09.128160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:34905 (local address 127.0.0.1:57398)
0808 18:31:09.128352 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:09.128406 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.130:34905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260808 18:31:09.137429 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 11637
I20260808 18:31:09.148447 30010 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 18:31:09.503438 30010 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 18:31:09.503575 30010 mini_ranger.cc:66] Stopping Ranger...
I20260808 18:31:09.848106 30010 mini_ranger.cc:68] Stopped Ranger
2026-08-08 18:31:09.848 UTC [10591] LOG: received smart shutdown request
2026-08-08 18:31:09.853 UTC [10591] LOG: background worker "logical replication launcher" (PID 10603) exited with exit code 1
2026-08-08 18:31:09.854 UTC [10596] LOG: shutting down
2026-08-08 18:31:10.047 UTC [10591] LOG: database system is shut down
2026-08-08T18:31:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50630 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:31:10 dist-test-slave-gp6n krb5kdc[12269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:31:10 dist-test-slave-gp6n krb5kdc[12269](info): set up 2 sockets
Aug 08 18:31:10 dist-test-slave-gp6n krb5kdc[12269](info): commencing operation
krb5kdc: starting...
W20260808 18:31:11.292057 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:31:11 dist-test-slave-gp6n krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:31:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:31:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:31:11.423724 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:40077
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:40077,127.29.78.189:39867,127.29.78.188:33455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:31:11.535291 12283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:11.535557 12283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:11.535629 12283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:11.539641 12283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:31:11.539712 12283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:11.539750 12283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:11.539805 12283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:31:11.539844 12283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:31:11.544981 12283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:40077,127.29.78.189:39867,127.29.78.188:33455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:40077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:11.546011 12283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:11.547088 12283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:11.555578 12290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:11.555547 12292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:11.555547 12289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:11.555583 12283 server_base.cc:1034] running on GCE node
I20260808 18:31:11.556329 12283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:11.557401 12283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:11.558591 12283 hybrid_clock.cc:648] HybridClock initialized: now 1786213871558581 us; error 27 us; skew 500 ppm
Aug 08 18:31:11 dist-test-slave-gp6n krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213871, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:11.561363 12283 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:31:11.562211 12283 webserver.cc:460] Webserver started at http://127.29.78.190:42795/ using document root <none> and password file <none>
I20260808 18:31:11.562908 12283 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:11.563054 12283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:11.563283 12283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:31:11.565310 12283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "f131401ad2e04900a0e9f08343d197d3"
format_stamp: "Formatted at 2026-08-08 18:31:11 on dist-test-slave-gp6n"
server_key: "60e70310c943e18cf98e13398a9a289c"
server_key_iv: "32b8e5fcbbdf4a9fe7c016dbd406e37b"
server_key_version: "encryptionkey@0"
I20260808 18:31:11.565877 12283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "f131401ad2e04900a0e9f08343d197d3"
format_stamp: "Formatted at 2026-08-08 18:31:11 on dist-test-slave-gp6n"
server_key: "60e70310c943e18cf98e13398a9a289c"
server_key_iv: "32b8e5fcbbdf4a9fe7c016dbd406e37b"
server_key_version: "encryptionkey@0"
I20260808 18:31:11.569484 12283 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 18:31:11.571959 12299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:11.573177 12283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:31:11.573315 12283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "f131401ad2e04900a0e9f08343d197d3"
format_stamp: "Formatted at 2026-08-08 18:31:11 on dist-test-slave-gp6n"
server_key: "60e70310c943e18cf98e13398a9a289c"
server_key_iv: "32b8e5fcbbdf4a9fe7c016dbd406e37b"
server_key_version: "encryptionkey@0"
I20260808 18:31:11.573426 12283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:11.593096 12283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:11.596043 12283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:11.596246 12283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:11.603749 12283 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:40077
I20260808 18:31:11.603773 12356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:40077 every 8 connection(s)
I20260808 18:31:11.604663 12283 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:31:11.607456 12357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:31:11.610455 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 12283
I20260808 18:31:11.610587 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:31:11.610932 30010 external_mini_cluster.cc:1428] Setting key 4acd293ae369cba6d3a43913a0b002b6
I20260808 18:31:11.611478 12357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40077 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.78.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.189@KRBTEST.COM" created.
Aug 08 18:31:11 dist-test-slave-gp6n krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213871, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:31:11.630342 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.614115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40077 (local address 127.0.0.1:46314)
0808 18:31:11.614719 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:11.614775 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:11.616033 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:11.616792 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:11.616801 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:11.617364 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:11.617782 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:11.617806 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:11.619074 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:11.619078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:11.620323 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:11.620335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:11.621017 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:11.621020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:11.621115 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:11.621724 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:11.621740 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:11.624898 (+ 3158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:11.627093 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:11.627106 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:11.627119 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:11.627450 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:11.627771 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:11.627777 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:11.627782 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:11.627938 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:11.628735 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:11.628743 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:11.628914 (+ 171us) client_negotiation.cc:769] Sending connection context
0808 18:31:11.629124 (+ 210us) client_negotiation.cc:240] Negotiation successful
0808 18:31:11.629621 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":113,"thread_start_us":140,"threads_started":1}
I20260808 18:31:11.630354 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.614115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46314 (local address 127.29.78.190:40077)
0808 18:31:11.614652 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:11.614668 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:11.614901 (+ 233us) server_negotiation.cc:407] Connection header received
0808 18:31:11.616391 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:11.616395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:11.616566 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:11.616713 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:11.617957 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:11.618938 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:11.620460 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:11.620868 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:11.620981 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:11.625073 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:11.625083 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:11.625098 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:11.625129 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:11.626979 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:11.627570 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:11.627575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:11.627576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:11.627676 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:11.628078 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:11.628081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:11.628082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:11.628596 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:11.628712 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:11.629234 (+ 522us) server_negotiation.cc:299] Negotiation successful
0808 18:31:11.629519 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":421,"spinlock_wait_cycles":41856,"thread_start_us":161,"threads_started":1}
I20260808 18:31:11.633031 12357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:31:11.633739 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.633501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39867 (local address 127.0.0.1:41126)
0808 18:31:11.633618 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:11.633687 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 18:31:11.633919 12304 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39867: connect: Connection refused (error 111)
I20260808 18:31:11.634907 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.634672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39867 (local address 127.0.0.1:41128)
0808 18:31:11.634801 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:11.634845 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 18:31:11.635061 12357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.189:39867: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39867: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.189.keytab.
Entry for principal kudu/127.29.78.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.189.keytab.
I20260808 18:31:11.641916 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.189:39867
--webserver_interface=127.29.78.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.189.keytab
--principal=kudu/127.29.78.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:40077,127.29.78.189:39867,127.29.78.188:33455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:31:11.684341 12357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } attempt: 1
I20260808 18:31:11.685218 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.684922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39867 (local address 127.0.0.1:41144)
0808 18:31:11.685070 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:11.685139 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 18:31:11.685963 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.685744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39867 (local address 127.0.0.1:41148)
0808 18:31:11.685854 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:11.685908 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 18:31:11.686093 12357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.189:39867: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39867: connect: Connection refused (error 111)
W20260808 18:31:11.752393 12365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:11.752681 12365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:11.752764 12365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 18:31:11.754278 12357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } attempt: 2
I20260808 18:31:11.755187 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.754870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39867 (local address 127.0.0.1:41152)
0808 18:31:11.755028 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:11.755106 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260808 18:31:11.755977 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.755742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39867 (local address 127.0.0.1:41158)
0808 18:31:11.755867 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:11.755925 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260808 18:31:11.756124 12357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.189:39867: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39867: connect: Connection refused (error 111)
W20260808 18:31:11.757113 12365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:31:11.757184 12365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:11.757231 12365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:11.757274 12365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:31:11.757309 12365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:31:11.762558 12365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:40077,127.29.78.189:39867,127.29.78.188:33455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.189.keytab
--principal=kudu/127.29.78.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.189:39867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.78.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:11.763561 12365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:11.764551 12365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:31:11.772925 12365 server_base.cc:1034] running on GCE node
W20260808 18:31:11.772913 12371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:11.772910 12372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:11.773092 12374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:11.773658 12365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:11.774761 12365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:11.775951 12365 hybrid_clock.cc:648] HybridClock initialized: now 1786213871775937 us; error 29 us; skew 500 ppm
Aug 08 18:31:11 dist-test-slave-gp6n krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213871, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:11.783844 12365 init.cc:377] Logged in from keytab as kudu/127.29.78.189@KRBTEST.COM (short username kudu)
I20260808 18:31:11.784788 12365 webserver.cc:460] Webserver started at http://127.29.78.189:45531/ using document root <none> and password file <none>
I20260808 18:31:11.785377 12365 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:11.785440 12365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:11.785660 12365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:31:11.787771 12365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-1/data/instance:
uuid: "d01391cd853448a5bbb6627e9e8de3bb"
format_stamp: "Formatted at 2026-08-08 18:31:11 on dist-test-slave-gp6n"
server_key: "75e12312b0cab76962b49a2c483e8dff"
server_key_iv: "1c7bc0797d1a6abaafda4bc8615a22c6"
server_key_version: "encryptionkey@0"
I20260808 18:31:11.788324 12365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-1/wal/instance:
uuid: "d01391cd853448a5bbb6627e9e8de3bb"
format_stamp: "Formatted at 2026-08-08 18:31:11 on dist-test-slave-gp6n"
server_key: "75e12312b0cab76962b49a2c483e8dff"
server_key_iv: "1c7bc0797d1a6abaafda4bc8615a22c6"
server_key_version: "encryptionkey@0"
I20260808 18:31:11.791939 12365 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 18:31:11.794243 12382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:11.795504 12365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:31:11.795626 12365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-1/wal
uuid: "d01391cd853448a5bbb6627e9e8de3bb"
format_stamp: "Formatted at 2026-08-08 18:31:11 on dist-test-slave-gp6n"
server_key: "75e12312b0cab76962b49a2c483e8dff"
server_key_iv: "1c7bc0797d1a6abaafda4bc8615a22c6"
server_key_version: "encryptionkey@0"
I20260808 18:31:11.795724 12365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:11.814654 12365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:11.819259 12365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:11.819509 12365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:11.829767 12365 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.189:39867
I20260808 18:31:11.829849 12437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.189:39867 every 8 connection(s)
I20260808 18:31:11.831103 12365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-1/data/info.pb
I20260808 18:31:11.835201 12438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:31:11.838671 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 12365
I20260808 18:31:11.838809 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-1/wal/instance
I20260808 18:31:11.839156 30010 external_mini_cluster.cc:1428] Setting key 5fcb09389ae09d43489eb0066214a7d5
I20260808 18:31:11.840739 12438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40077 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:31:11.847285 12357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } attempt: 3
WARNING: no policy specified for kudu/127.29.78.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.188@KRBTEST.COM" created.
Aug 08 18:31:11 dist-test-slave-gp6n krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213871, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
Aug 08 18:31:11 dist-test-slave-gp6n krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213871, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
I20260808 18:31:11.865639 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.843132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46318 (local address 127.29.78.190:40077)
0808 18:31:11.844191 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:11.844197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:11.844519 (+ 322us) server_negotiation.cc:407] Connection header received
0808 18:31:11.845901 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:11.845907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:11.845985 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:11.846105 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:11.848684 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:11.849549 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:11.850939 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:11.851196 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:11.851313 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:11.858813 (+ 7500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:11.858819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:11.858823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:11.858855 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:11.861835 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:11.862583 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:11.862589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:11.862593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:11.862695 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:11.863160 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:11.863166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:11.863169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:11.863470 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:11.863576 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:11.865230 (+ 1654us) server_negotiation.cc:299] Negotiation successful
0808 18:31:11.865405 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":932}
I20260808 18:31:11.867559 12441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.843734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40077 (local address 127.0.0.1:46318)
0808 18:31:11.844319 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:11.844379 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:11.845603 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:11.846545 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:11.846559 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:11.847177 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:11.848510 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:11.848532 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:11.849706 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:11.849709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:11.850804 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:11.850817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:11.851448 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:11.851453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:11.851701 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:11.856436 (+ 4735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:11.856455 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:11.858614 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:11.861969 (+ 3355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:11.861978 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:11.861997 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:11.862440 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:11.862823 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:11.862826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:11.862828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:11.863031 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:11.863657 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:11.863666 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:11.864033 (+ 367us) client_negotiation.cc:769] Sending connection context
0808 18:31:11.865905 (+ 1872us) client_negotiation.cc:240] Negotiation successful
0808 18:31:11.866309 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":159,"threads_started":1}
I20260808 18:31:11.868726 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.847802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39867 (local address 127.0.0.1:41172)
0808 18:31:11.848100 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:11.848116 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:11.848203 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:11.858866 (+ 10663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:11.858869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:11.859073 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:11.859202 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:11.859208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:11.860316 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:11.860325 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:11.861236 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:11.861250 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:11.862132 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:11.862135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:11.862266 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:11.862766 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:11.862786 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:11.864405 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:11.866876 (+ 2471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:11.866881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:11.866885 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:11.867159 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:11.867516 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:11.867519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:11.867521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:11.867654 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:11.868179 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:11.868182 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:11.868310 (+ 128us) client_negotiation.cc:769] Sending connection context
0808 18:31:11.868426 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 18:31:11.868561 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221}
I20260808 18:31:11.868844 12443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.849318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41172 (local address 127.29.78.189:39867)
0808 18:31:11.858341 (+ 9023us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:11.858351 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:11.858397 (+ 46us) server_negotiation.cc:407] Connection header received
0808 18:31:11.858476 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:11.858482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:11.858712 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:11.859336 (+ 624us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:11.859427 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:11.860199 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:11.861386 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:11.862005 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:11.862132 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:11.864568 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:11.864581 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:11.864599 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:11.864628 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:11.866752 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:11.867314 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:11.867320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:11.867322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:11.867406 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:11.867781 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:11.867785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:11.867786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:11.868064 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:11.868211 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:11.868542 (+ 331us) server_negotiation.cc:299] Negotiation successful
0808 18:31:11.868676 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8896,"thread_start_us":158,"threads_started":1}
I20260808 18:31:11.869839 12438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:31:11.871074 12357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:31:11.871742 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.871521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33455 (local address 127.0.0.1:35456)
0808 18:31:11.871628 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:11.871679 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 18:31:11.872421 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.872182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33455 (local address 127.0.0.1:35472)
0808 18:31:11.872337 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:11.872379 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260808 18:31:11.872588 12357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:33455: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
Aug 08 18:31:11 dist-test-slave-gp6n krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213871, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.188.keytab.
Entry for principal kudu/127.29.78.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.188.keytab.
I20260808 18:31:11.877305 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.188:33455
--webserver_interface=127.29.78.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.188.keytab
--principal=kudu/127.29.78.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:40077,127.29.78.189:39867,127.29.78.188:33455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:31:11.878505 12441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.870446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39867 (local address 127.0.0.1:41178)
0808 18:31:11.870563 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:11.870575 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:11.870637 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:11.870895 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:11.870897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:11.871053 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:11.871175 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:11.871182 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:11.872046 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:11.872051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:11.872884 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:11.872893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:11.873427 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:11.873430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:11.873504 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:11.873897 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:11.873908 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:11.875241 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:11.876945 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:11.876948 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:11.876950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:11.877187 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:11.877481 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:11.877484 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:11.877486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:11.877563 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:11.877957 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:11.877960 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:11.878039 (+ 79us) client_negotiation.cc:769] Sending connection context
0808 18:31:11.878151 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 18:31:11.878283 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260808 18:31:11.878505 12443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.870599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41178 (local address 127.29.78.189:39867)
0808 18:31:11.870716 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:11.870722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:11.870736 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:31:11.870772 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:11.870775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:11.870811 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:11.870880 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:11.871320 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:11.871933 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:11.872993 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:11.873305 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:11.873409 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:11.875398 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:11.875401 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:11.875402 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:11.875422 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:11.876823 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:11.877281 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:11.877286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:11.877288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:11.877387 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:11.877654 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:11.877657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:11.877659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:11.877859 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:11.877957 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:11.878220 (+ 263us) server_negotiation.cc:299] Negotiation successful
0808 18:31:11.878341 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":52}
I20260808 18:31:11.879221 12438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:31:11.879940 12441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.879655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33455 (local address 127.0.0.1:35484)
0808 18:31:11.879778 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:11.879865 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 18:31:11.880712 12385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
I20260808 18:31:11.881590 12441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.881389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33455 (local address 127.0.0.1:35494)
0808 18:31:11.881503 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:11.881544 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 18:31:11.881798 12438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:33455: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
I20260808 18:31:11.903771 12357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } attempt: 1
I20260808 18:31:11.904489 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.904222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33455 (local address 127.0.0.1:35504)
0808 18:31:11.904356 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:11.904418 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 18:31:11.905169 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.904953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33455 (local address 127.0.0.1:35516)
0808 18:31:11.905065 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:11.905112 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 18:31:11.905292 12357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:33455: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
I20260808 18:31:11.931039 12438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } attempt: 1
I20260808 18:31:11.931883 12441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.931594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33455 (local address 127.0.0.1:35528)
0808 18:31:11.931746 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:11.931807 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 18:31:11.932614 12441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.932385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33455 (local address 127.0.0.1:35530)
0808 18:31:11.932523 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:11.932566 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 18:31:11.932744 12438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:33455: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
I20260808 18:31:11.980440 12357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } attempt: 2
I20260808 18:31:11.981334 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.981032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33455 (local address 127.0.0.1:35536)
0808 18:31:11.981186 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:11.981251 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 18:31:11.982074 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:11.981860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33455 (local address 127.0.0.1:35552)
0808 18:31:11.981966 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:11.982025 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 18:31:11.982206 12357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:33455: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
W20260808 18:31:11.995167 12445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:11.995494 12445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:11.995609 12445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:12.000141 12445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:31:12.000226 12445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:12.000274 12445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:12.000344 12445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:31:12.000389 12445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:31:12.000890 12438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } attempt: 2
I20260808 18:31:12.001711 12441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:12.001428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33455 (local address 127.0.0.1:35560)
0808 18:31:12.001576 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:12.001640 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 18:31:12.002388 12441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:12.002184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33455 (local address 127.0.0.1:35572)
0808 18:31:12.002291 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:12.002340 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 18:31:12.002532 12438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:33455: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
I20260808 18:31:12.006629 12445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:40077,127.29.78.189:39867,127.29.78.188:33455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.188.keytab
--principal=kudu/127.29.78.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.188:33455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.78.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:12.007732 12445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:12.008824 12445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:12.017823 12455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:12.017822 12452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:12.017829 12451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:12.018193 12445 server_base.cc:1034] running on GCE node
I20260808 18:31:12.018846 12445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:12.019887 12445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:12.021099 12445 hybrid_clock.cc:648] HybridClock initialized: now 1786213872021064 us; error 43 us; skew 500 ppm
Aug 08 18:31:12 dist-test-slave-gp6n krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213872, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:12.024010 12445 init.cc:377] Logged in from keytab as kudu/127.29.78.188@KRBTEST.COM (short username kudu)
I20260808 18:31:12.024940 12445 webserver.cc:460] Webserver started at http://127.29.78.188:34953/ using document root <none> and password file <none>
I20260808 18:31:12.025483 12445 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:12.025574 12445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:12.025800 12445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:31:12.027806 12445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-2/data/instance:
uuid: "0e8b2530335a45b2a521468b702479bb"
format_stamp: "Formatted at 2026-08-08 18:31:12 on dist-test-slave-gp6n"
server_key: "5e58306d04cd83c4f5d2bf0cbbad9c38"
server_key_iv: "f4554e88342477fa031f89ddb86a0495"
server_key_version: "encryptionkey@0"
I20260808 18:31:12.028404 12445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-2/wal/instance:
uuid: "0e8b2530335a45b2a521468b702479bb"
format_stamp: "Formatted at 2026-08-08 18:31:12 on dist-test-slave-gp6n"
server_key: "5e58306d04cd83c4f5d2bf0cbbad9c38"
server_key_iv: "f4554e88342477fa031f89ddb86a0495"
server_key_version: "encryptionkey@0"
I20260808 18:31:12.032294 12445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:31:12.034662 12462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:12.035801 12445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:31:12.035964 12445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-2/wal
uuid: "0e8b2530335a45b2a521468b702479bb"
format_stamp: "Formatted at 2026-08-08 18:31:12 on dist-test-slave-gp6n"
server_key: "5e58306d04cd83c4f5d2bf0cbbad9c38"
server_key_iv: "f4554e88342477fa031f89ddb86a0495"
server_key_version: "encryptionkey@0"
I20260808 18:31:12.036088 12445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:12.081369 12357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } attempt: 3
I20260808 18:31:12.082242 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:12.081940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33455 (local address 127.0.0.1:35582)
0808 18:31:12.082091 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:12.082156 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 18:31:12.082993 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:12.082781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33455 (local address 127.0.0.1:35590)
0808 18:31:12.082888 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:12.082939 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 18:31:12.083124 12357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:33455: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
I20260808 18:31:12.087906 12445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:12.091892 12445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:12.092127 12445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:12.093683 12438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } attempt: 3
I20260808 18:31:12.094363 12441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:12.094109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33455 (local address 127.0.0.1:35602)
0808 18:31:12.094236 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:12.094295 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 18:31:12.095139 12441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:12.094889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33455 (local address 127.0.0.1:35618)
0808 18:31:12.095042 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:12.095094 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260808 18:31:12.095253 12438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:33455: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33455: connect: Connection refused (error 111)
I20260808 18:31:12.100370 12445 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.188:33455
I20260808 18:31:12.100369 12518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.188:33455 every 8 connection(s)
I20260808 18:31:12.101425 12445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-2/data/info.pb
I20260808 18:31:12.104267 12519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:31:12.105417 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 12445
I20260808 18:31:12.105551 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786213708769829-30010-0/minicluster-data/master-2/wal/instance
I20260808 18:31:12.105834 30010 external_mini_cluster.cc:1428] Setting key 74721a472ee7a9eedff895269187b612
I20260808 18:31:12.109217 12519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40077 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 18:31:12 dist-test-slave-gp6n krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
Aug 08 18:31:12 dist-test-slave-gp6n krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213872, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:31:12.120725 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:12.107173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.29.78.188:33455)
0808 18:31:12.107682 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:12.107701 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:12.107749 (+ 48us) server_negotiation.cc:407] Connection header received
0808 18:31:12.108557 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:12.108576 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:12.108905 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:12.109183 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:12.110118 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.111270 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.112269 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.112931 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.113044 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:12.116024 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:12.116029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:12.116038 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:12.116060 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:12.117957 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:12.118373 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:12.118379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:12.118385 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:12.118534 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:12.118822 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:12.118826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:12.118828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:12.119201 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:12.119398 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:12.119743 (+ 345us) server_negotiation.cc:299] Negotiation successful
0808 18:31:12.120035 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"spinlock_wait_cycles":1920,"thread_start_us":161,"threads_started":1}
I20260808 18:31:12.120831 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:12.111200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46328 (local address 127.29.78.190:40077)
0808 18:31:12.111352 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:12.111357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:12.111632 (+ 275us) server_negotiation.cc:407] Connection header received
0808 18:31:12.111814 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:12.111817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:12.111869 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:12.111958 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:12.112712 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.113323 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.114330 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.114721 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.114806 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:12.117278 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:12.117282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:12.117284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:12.117307 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:12.118781 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:12.119278 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:12.119286 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:12.119287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:12.119369 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:12.119799 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:12.119803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:12.119805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:12.120033 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:12.120157 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:12.120534 (+ 377us) server_negotiation.cc:299] Negotiation successful
0808 18:31:12.120652 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 18:31:12.120867 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:12.111134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40077 (local address 127.0.0.1:46328)
0808 18:31:12.111514 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:12.111553 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:12.111713 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:12.112006 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:12.112011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:12.112444 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:12.112585 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:12.112592 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.113469 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.113473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:12.114204 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:12.114213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.114891 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.114896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:12.115039 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:12.115635 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:12.115661 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:12.117133 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:12.118909 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:12.118919 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:12.118924 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:12.119160 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:12.119577 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:12.119583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:12.119585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:12.119691 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:12.120157 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:12.120163 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:12.120304 (+ 141us) client_negotiation.cc:769] Sending connection context
0808 18:31:12.120460 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 18:31:12.120649 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":148,"threads_started":1}
I20260808 18:31:12.122035 12519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 18:31:12 dist-test-slave-gp6n krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213872, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
Aug 08 18:31:12 dist-test-slave-gp6n krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:31:12.130119 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:12.122585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39867 (local address 127.0.0.1:41184)
0808 18:31:12.122844 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:12.122854 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:12.122918 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:12.123170 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:12.123173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:12.123348 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:12.123427 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:12.123432 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.124135 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.124140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:12.124635 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:12.124642 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.125111 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.125115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:12.125196 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:12.125656 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:12.125666 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:12.126835 (+ 1169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:12.128663 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:12.128666 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:12.128668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:12.128859 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:12.129157 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:12.129159 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:12.129161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:12.129245 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:12.129623 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:12.129626 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:12.129719 (+ 93us) client_negotiation.cc:769] Sending connection context
0808 18:31:12.129839 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 18:31:12.129961 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260808 18:31:12.130151 12443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:12.122742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41184 (local address 127.29.78.189:39867)
0808 18:31:12.122924 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:12.122932 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:12.122950 (+ 18us) server_negotiation.cc:407] Connection header received
0808 18:31:12.122999 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:12.123002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:12.123056 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:12.123148 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:12.123541 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.124028 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.124750 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.124996 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.125100 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:12.126978 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:12.126981 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:12.126983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:12.127004 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:12.128545 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:12.128958 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:12.128962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:12.128964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:12.129055 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:12.129346 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:12.129351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:12.129353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:12.129524 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:12.129631 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:12.129900 (+ 269us) server_negotiation.cc:299] Negotiation successful
0808 18:31:12.130008 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 18:31:12.130828 12519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:31:12.132387 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:12.123273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46340 (local address 127.29.78.190:40077)
0808 18:31:12.123371 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:12.123375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:12.123387 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:31:12.123428 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:12.123430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:12.123462 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:12.123560 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:12.124216 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.124742 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.125684 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.125938 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.126055 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:12.128813 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:12.128817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:12.128819 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:12.128841 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:12.130581 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:12.131129 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:12.131133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:12.131135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:12.131229 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:12.131554 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:12.131558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:12.131560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:12.131741 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:12.132013 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:12.132104 (+ 91us) server_negotiation.cc:299] Negotiation successful
0808 18:31:12.132219 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
Aug 08 18:31:12 dist-test-slave-gp6n krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213872, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
Aug 08 18:31:12 dist-test-slave-gp6n krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
I20260808 18:31:12.139889 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:12.131401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35630 (local address 127.29.78.188:33455)
0808 18:31:12.131548 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:12.131554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:12.131813 (+ 259us) server_negotiation.cc:407] Connection header received
0808 18:31:12.131930 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:12.131933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:12.131964 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:12.132053 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:12.132407 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.133132 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.134097 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.134346 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.134537 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:12.136418 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:12.136422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:12.136425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:12.136447 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:12.137938 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:12.138328 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:12.138332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:12.138334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:12.138454 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:12.138773 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:12.138780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:12.138785 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:12.139020 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:12.139167 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:12.139612 (+ 445us) server_negotiation.cc:299] Negotiation successful
0808 18:31:12.139752 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":52}
I20260808 18:31:12.140043 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:12.131661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33455 (local address 127.0.0.1:35630)
0808 18:31:12.131753 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:12.131767 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:12.131847 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:12.132062 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:12.132065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:12.132217 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:12.132268 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:12.132273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.133258 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.133262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:12.133972 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:12.133981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.134538 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.134542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:12.134627 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:12.135059 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:12.135070 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:12.136274 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:12.138052 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:12.138054 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:12.138056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:12.138230 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:12.138556 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:12.138564 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:12.138566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:12.138661 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:12.139205 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:12.139208 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:12.139320 (+ 112us) client_negotiation.cc:769] Sending connection context
0808 18:31:12.139514 (+ 194us) client_negotiation.cc:240] Negotiation successful
0808 18:31:12.139680 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260808 18:31:12.142886 12443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:12.133627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41198 (local address 127.29.78.189:39867)
0808 18:31:12.133740 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:12.133745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:12.133757 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:31:12.133851 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:12.133854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:12.133897 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:12.133986 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:12.134634 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.135267 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.136025 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.136274 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.136378 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:12.139222 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:12.139226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:12.139229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:12.139256 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:12.141174 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:12.141684 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:12.141687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:12.141689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:12.141768 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:12.142066 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:12.142068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:12.142069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:12.142234 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:12.142553 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:12.142624 (+ 71us) server_negotiation.cc:299] Negotiation successful
0808 18:31:12.142727 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 18:31:12.143695 12519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb: Bootstrap starting.
I20260808 18:31:12.146270 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:31:12.146512 12519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb: Neither blocks nor log segments found. Creating new log.
I20260808 18:31:12.147291 12519 log.cc:824] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:12.149681 12519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb: No bootstrap required, opened a new log
I20260808 18:31:12.155017 12519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f131401ad2e04900a0e9f08343d197d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40077 } } peers { permanent_uuid: "d01391cd853448a5bbb6627e9e8de3bb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } } peers { permanent_uuid: "0e8b2530335a45b2a521468b702479bb" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } }
I20260808 18:31:12.155956 12519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:31:12.156677 12519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e8b2530335a45b2a521468b702479bb, State: Initialized, Role: FOLLOWER
I20260808 18:31:12.157334 12519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f131401ad2e04900a0e9f08343d197d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40077 } } peers { permanent_uuid: "d01391cd853448a5bbb6627e9e8de3bb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } } peers { permanent_uuid: "0e8b2530335a45b2a521468b702479bb" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } }
I20260808 18:31:12.158921 12519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:31:12.159155 12443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:12.148470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41204 (local address 127.29.78.189:39867)
0808 18:31:12.148807 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:12.148813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:12.149439 (+ 626us) server_negotiation.cc:407] Connection header received
0808 18:31:12.149609 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:12.149615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:12.149661 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:12.149729 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:12.150693 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.151404 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.152480 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.152819 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.152899 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:12.155066 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:12.155070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:12.155072 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:12.155097 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:12.157160 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:12.157745 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:12.157749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:12.157750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:12.157842 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:12.158173 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:12.158179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:12.158184 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:12.158389 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:12.158510 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:12.158901 (+ 391us) server_negotiation.cc:299] Negotiation successful
0808 18:31:12.159005 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268}
I20260808 18:31:12.159155 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:12.148685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.29.78.188:33455)
0808 18:31:12.148795 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:12.148799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:12.149086 (+ 287us) server_negotiation.cc:407] Connection header received
0808 18:31:12.149261 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:12.149264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:12.149332 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:12.149418 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:12.150523 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.151268 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.152480 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.152821 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.152896 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:12.155066 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:12.155070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:12.155072 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:12.155097 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:12.157134 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:12.157857 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:12.157860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:12.157862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:12.157946 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:12.158251 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:12.158254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:12.158256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:12.158509 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:12.158599 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:12.158892 (+ 293us) server_negotiation.cc:299] Negotiation successful
0808 18:31:12.159011 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 18:31:12.159533 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:12.148694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46344 (local address 127.29.78.190:40077)
0808 18:31:12.148797 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:12.148802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:12.149126 (+ 324us) server_negotiation.cc:407] Connection header received
0808 18:31:12.149176 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:12.149179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:12.149229 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:12.149336 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:12.150344 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.151158 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.152673 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.153052 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.153134 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:12.155292 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:12.155296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:12.155299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:12.155333 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:12.157435 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:12.158028 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:12.158032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:12.158034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:12.158126 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:12.158507 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:12.158510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:12.158512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:12.158729 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:12.158829 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:12.159243 (+ 414us) server_negotiation.cc:299] Negotiation successful
0808 18:31:12.159377 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 18:31:12.160126 12532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f131401ad2e04900a0e9f08343d197d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40077 } } peers { permanent_uuid: "d01391cd853448a5bbb6627e9e8de3bb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } } peers { permanent_uuid: "0e8b2530335a45b2a521468b702479bb" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } } }
I20260808 18:31:12.160276 12532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb [sys.catalog]: This master's current role is: FOLLOWER
W20260808 18:31:12.164579 12544 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:31:12.164656 12544 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:31:12.162736 12445 master_runner.cc:186] Error getting master registration for 127.29.78.190:40077: OK, instance_id { permanent_uuid: "f131401ad2e04900a0e9f08343d197d3" instance_seqno: 1786213871601591 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 18:31:12.165720 12445 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 18:31:12.238440 12438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } attempt: 4
Aug 08 18:31:12 dist-test-slave-gp6n krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213871, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
I20260808 18:31:12.247282 12357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } attempt: 4
I20260808 18:31:12.247803 12441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:12.238981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33455 (local address 127.0.0.1:35634)
0808 18:31:12.239131 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:12.239151 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:12.239242 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:12.239515 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:12.239518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:12.239763 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:12.239857 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:12.239863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.240735 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.240740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:12.241328 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:12.241343 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.241822 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.241826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:12.241944 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:12.242580 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:12.242593 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:12.244201 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:12.246046 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:12.246050 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:12.246052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:12.246334 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:12.246666 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:12.246670 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:12.246672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:12.246806 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:12.247246 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:12.247250 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:12.247359 (+ 109us) client_negotiation.cc:769] Sending connection context
0808 18:31:12.247483 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 18:31:12.247623 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 18:31:12.248273 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:12.239072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35634 (local address 127.29.78.188:33455)
0808 18:31:12.239233 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:12.239238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:12.239254 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:31:12.239343 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:12.239349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:12.239394 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:12.239496 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:12.239980 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.240617 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.241462 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.241704 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.241821 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:12.244358 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:12.244362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:12.244418 (+ 56us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:12.244449 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:12.245929 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:12.246486 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:12.246489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:12.246490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:12.246564 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:12.246907 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:12.246909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:12.246911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:12.247152 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:12.247230 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:12.247562 (+ 332us) server_negotiation.cc:299] Negotiation successful
0808 18:31:12.247689 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 18:31:12.251463 12438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb: Bootstrap starting.
Aug 08 18:31:12 dist-test-slave-gp6n krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213871, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
I20260808 18:31:12.253937 12438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb: Neither blocks nor log segments found. Creating new log.
I20260808 18:31:12.254916 12438 log.cc:824] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:12.257369 12438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb: No bootstrap required, opened a new log
I20260808 18:31:12.257717 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:12.247674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33455 (local address 127.0.0.1:35638)
0808 18:31:12.247843 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:12.247859 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:12.247945 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:12.248957 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:12.248960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:12.249169 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:12.249279 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:12.249287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.250305 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.250309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:12.251260 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:12.251271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.251855 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.251859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:12.251958 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:12.252437 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:12.252452 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:12.254136 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:12.256258 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:12.256261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:12.256263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:12.256464 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:12.256752 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:12.256756 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:12.256758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:12.256842 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:12.257219 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:12.257222 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:12.257315 (+ 93us) client_negotiation.cc:769] Sending connection context
0808 18:31:12.257426 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 18:31:12.257564 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260808 18:31:12.257735 12546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:12.248149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35638 (local address 127.29.78.188:33455)
0808 18:31:12.248693 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:12.248700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:12.248713 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:31:12.248747 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:12.248765 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:12.248812 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:12.248911 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:12.249432 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.250179 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.251402 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:12.251732 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:12.251836 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:12.254287 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:12.254290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:12.254293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:12.254317 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:12.256138 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:12.256567 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:12.256571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:12.256572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:12.256660 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:12.256936 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:12.256940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:12.256942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:12.257124 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:12.257225 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:12.257477 (+ 252us) server_negotiation.cc:299] Negotiation successful
0808 18:31:12.257578 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":156,"threads_started":1}
I20260808 18:31:12.261102 12357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3: Bootstrap starting.
I20260808 18:31:12.263180 12357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3: Neither blocks nor log segments found. Creating new log.
I20260808 18:31:12.263217 12438 raft_consensus.cc:348] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f131401ad2e04900a0e9f08343d197d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40077 } } peers { permanent_uuid: "d01391cd853448a5bbb6627e9e8de3bb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } } peers { permanent_uuid: "0e8b2530335a45b2a521468b702479bb" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } }
I20260808 18:31:12.263458 12438 raft_consensus.cc:374] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:31:12.263520 12438 raft_consensus.cc:729] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d01391cd853448a5bbb6627e9e8de3bb, State: Initialized, Role: FOLLOWER
I20260808 18:31:12.264050 12357 log.cc:824] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:12.264000 12438 consensus_queue.cc:260] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f131401ad2e04900a0e9f08343d197d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40077 } } peers { permanent_uuid: "d01391cd853448a5bbb6627e9e8de3bb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } } peers { permanent_uuid: "0e8b2530335a45b2a521468b702479bb" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } }
I20260808 18:31:12.264531 12549 sys_catalog.cc:455] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f131401ad2e04900a0e9f08343d197d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40077 } } peers { permanent_uuid: "d01391cd853448a5bbb6627e9e8de3bb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } } peers { permanent_uuid: "0e8b2530335a45b2a521468b702479bb" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } } }
I20260808 18:31:12.264742 12549 sys_catalog.cc:458] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:31:12.265369 12438 sys_catalog.cc:564] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:31:12.266489 12357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3: No bootstrap required, opened a new log
W20260808 18:31:12.270496 12562 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:31:12.270627 12562 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:31:12.270545 12357 raft_consensus.cc:348] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f131401ad2e04900a0e9f08343d197d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40077 } } peers { permanent_uuid: "d01391cd853448a5bbb6627e9e8de3bb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } } peers { permanent_uuid: "0e8b2530335a45b2a521468b702479bb" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } }
I20260808 18:31:12.270777 12357 raft_consensus.cc:374] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:31:12.270836 12357 raft_consensus.cc:729] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f131401ad2e04900a0e9f08343d197d3, State: Initialized, Role: FOLLOWER
I20260808 18:31:12.271337 12357 consensus_queue.cc:260] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f131401ad2e04900a0e9f08343d197d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40077 } } peers { permanent_uuid: "d01391cd853448a5bbb6627e9e8de3bb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } } peers { permanent_uuid: "0e8b2530335a45b2a521468b702479bb" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } }
I20260808 18:31:12.272637 12357 sys_catalog.cc:564] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:31:12.268793 12365 master_runner.cc:186] Error getting master registration for 127.29.78.190:40077: OK, instance_id { permanent_uuid: "f131401ad2e04900a0e9f08343d197d3" instance_seqno: 1786213871601591 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 18:31:12.273070 12365 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 18:31:12.275038 12565 sys_catalog.cc:455] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f131401ad2e04900a0e9f08343d197d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40077 } } peers { permanent_uuid: "d01391cd853448a5bbb6627e9e8de3bb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } } peers { permanent_uuid: "0e8b2530335a45b2a521468b702479bb" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } } }
I20260808 18:31:12.277038 12565 sys_catalog.cc:458] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 18:31:12.279515 12577 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:31:12.279594 12577 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:31:12.281994 12283 master_runner.cc:263] No leader master found from master f131401ad2e04900a0e9f08343d197d3
I20260808 18:31:12.282074 12283 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 18:31:12.321614 12388 tablet.cc:2369] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:31:12.322573 12565 raft_consensus.cc:482] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:31:12.322750 12565 raft_consensus.cc:504] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f131401ad2e04900a0e9f08343d197d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40077 } } peers { permanent_uuid: "d01391cd853448a5bbb6627e9e8de3bb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } } peers { permanent_uuid: "0e8b2530335a45b2a521468b702479bb" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } }
I20260808 18:31:12.323848 12565 leader_election.cc:288] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d01391cd853448a5bbb6627e9e8de3bb (127.29.78.189:39867), 0e8b2530335a45b2a521468b702479bb (127.29.78.188:33455)
I20260808 18:31:12.324527 12410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f131401ad2e04900a0e9f08343d197d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d01391cd853448a5bbb6627e9e8de3bb" is_pre_election: true
I20260808 18:31:12.324829 12410 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f131401ad2e04900a0e9f08343d197d3 in term 0.
I20260808 18:31:12.325464 12304 leader_election.cc:302] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d01391cd853448a5bbb6627e9e8de3bb, f131401ad2e04900a0e9f08343d197d3; no voters:
I20260808 18:31:12.326283 12490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f131401ad2e04900a0e9f08343d197d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e8b2530335a45b2a521468b702479bb" is_pre_election: true
I20260808 18:31:12.326609 12490 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f131401ad2e04900a0e9f08343d197d3 in term 0.
I20260808 18:31:12.327287 12565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:31:12.327379 12565 raft_consensus.cc:482] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:31:12.327427 12565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:31:12.328908 12565 raft_consensus.cc:504] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f131401ad2e04900a0e9f08343d197d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40077 } } peers { permanent_uuid: "d01391cd853448a5bbb6627e9e8de3bb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } } peers { permanent_uuid: "0e8b2530335a45b2a521468b702479bb" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } }
I20260808 18:31:12.329262 12565 leader_election.cc:288] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [CANDIDATE]: Term 1 election: Requested vote from peers d01391cd853448a5bbb6627e9e8de3bb (127.29.78.189:39867), 0e8b2530335a45b2a521468b702479bb (127.29.78.188:33455)
I20260808 18:31:12.330049 12490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f131401ad2e04900a0e9f08343d197d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e8b2530335a45b2a521468b702479bb"
I20260808 18:31:12.330171 12490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:31:12.330575 12410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f131401ad2e04900a0e9f08343d197d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d01391cd853448a5bbb6627e9e8de3bb"
I20260808 18:31:12.330698 12410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:31:12.332032 12410 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f131401ad2e04900a0e9f08343d197d3 in term 1.
I20260808 18:31:12.332434 12304 leader_election.cc:302] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d01391cd853448a5bbb6627e9e8de3bb, f131401ad2e04900a0e9f08343d197d3; no voters:
I20260808 18:31:12.333894 12490 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f131401ad2e04900a0e9f08343d197d3 in term 1.
I20260808 18:31:12.336879 12565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:31:12.337186 12565 raft_consensus.cc:686] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [term 1 LEADER]: Becoming Leader. State: Replica: f131401ad2e04900a0e9f08343d197d3, State: Running, Role: LEADER
I20260808 18:31:12.337604 12565 consensus_queue.cc:237] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f131401ad2e04900a0e9f08343d197d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40077 } } peers { permanent_uuid: "d01391cd853448a5bbb6627e9e8de3bb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } } peers { permanent_uuid: "0e8b2530335a45b2a521468b702479bb" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } }
I20260808 18:31:12.339671 12565 sys_catalog.cc:455] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f131401ad2e04900a0e9f08343d197d3. Latest consensus state: current_term: 1 leader_uuid: "f131401ad2e04900a0e9f08343d197d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f131401ad2e04900a0e9f08343d197d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40077 } } peers { permanent_uuid: "d01391cd853448a5bbb6627e9e8de3bb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } } peers { permanent_uuid: "0e8b2530335a45b2a521468b702479bb" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } } }
I20260808 18:31:12.339864 12565 sys_catalog.cc:458] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [sys.catalog]: This master's current role is: LEADER
I20260808 18:31:12.340371 12581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:31:12.341429 12581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260808 18:31:12.343639 12468 tablet.cc:2369] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 18:31:12.347982 12306 tablet.cc:2369] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:31:12.349627 12490 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb [term 1 FOLLOWER]: Refusing update from remote peer f131401ad2e04900a0e9f08343d197d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:31:12.350490 12565 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e8b2530335a45b2a521468b702479bb" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:31:12.357630 12532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f131401ad2e04900a0e9f08343d197d3. Latest consensus state: current_term: 1 leader_uuid: "f131401ad2e04900a0e9f08343d197d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f131401ad2e04900a0e9f08343d197d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40077 } } peers { permanent_uuid: "d01391cd853448a5bbb6627e9e8de3bb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } } peers { permanent_uuid: "0e8b2530335a45b2a521468b702479bb" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } } }
I20260808 18:31:12.357759 12532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:31:12.359309 12410 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb [term 1 FOLLOWER]: Refusing update from remote peer f131401ad2e04900a0e9f08343d197d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:31:12.360008 12565 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d01391cd853448a5bbb6627e9e8de3bb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:31:12.364369 12549 sys_catalog.cc:455] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f131401ad2e04900a0e9f08343d197d3. Latest consensus state: current_term: 1 leader_uuid: "f131401ad2e04900a0e9f08343d197d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f131401ad2e04900a0e9f08343d197d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40077 } } peers { permanent_uuid: "d01391cd853448a5bbb6627e9e8de3bb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } } peers { permanent_uuid: "0e8b2530335a45b2a521468b702479bb" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } } }
I20260808 18:31:12.364506 12549 sys_catalog.cc:458] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:31:12.366374 12582 mvcc.cc:205] Tried to move back new op lower bound from 7316332021134503936 to 7316332021098565632. Current Snapshot: MvccSnapshot[applied={T|T < 7316332021134503936}]
I20260808 18:31:12.366122 12579 sys_catalog.cc:455] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f131401ad2e04900a0e9f08343d197d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f131401ad2e04900a0e9f08343d197d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40077 } } peers { permanent_uuid: "d01391cd853448a5bbb6627e9e8de3bb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } } peers { permanent_uuid: "0e8b2530335a45b2a521468b702479bb" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } } }
I20260808 18:31:12.366675 12579 sys_catalog.cc:458] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [sys.catalog]: This master's current role is: LEADER
I20260808 18:31:12.367961 12532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f131401ad2e04900a0e9f08343d197d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f131401ad2e04900a0e9f08343d197d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40077 } } peers { permanent_uuid: "d01391cd853448a5bbb6627e9e8de3bb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } } peers { permanent_uuid: "0e8b2530335a45b2a521468b702479bb" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } } }
I20260808 18:31:12.368074 12532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:31:12.369160 12565 sys_catalog.cc:455] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f131401ad2e04900a0e9f08343d197d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f131401ad2e04900a0e9f08343d197d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40077 } } peers { permanent_uuid: "d01391cd853448a5bbb6627e9e8de3bb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } } peers { permanent_uuid: "0e8b2530335a45b2a521468b702479bb" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } } }
I20260808 18:31:12.369273 12565 sys_catalog.cc:458] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3 [sys.catalog]: This master's current role is: LEADER
I20260808 18:31:12.369769 12549 sys_catalog.cc:455] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f131401ad2e04900a0e9f08343d197d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f131401ad2e04900a0e9f08343d197d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40077 } } peers { permanent_uuid: "d01391cd853448a5bbb6627e9e8de3bb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39867 } } peers { permanent_uuid: "0e8b2530335a45b2a521468b702479bb" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33455 } } }
I20260808 18:31:12.369907 12549 sys_catalog.cc:458] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:31:12.372924 12581 catalog_manager.cc:1347] Generated new cluster ID: 1096b94dba744443980f6999955d189b
I20260808 18:31:12.372989 12581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:31:12.409236 12581 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:31:12.410826 12581 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:31:12.434113 12581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f131401ad2e04900a0e9f08343d197d3: Generated new TSK 0
I20260808 18:31:12.435671 12581 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:31:13.166143 12544 catalog_manager.cc:1259] Loaded cluster ID: 1096b94dba744443980f6999955d189b
I20260808 18:31:13.166227 12544 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb: loading cluster ID for follower catalog manager: success
I20260808 18:31:13.167958 12544 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb: acquiring CA information for follower catalog manager: success
I20260808 18:31:13.169114 12544 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0e8b2530335a45b2a521468b702479bb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 18:31:13.271994 12562 catalog_manager.cc:1259] Loaded cluster ID: 1096b94dba744443980f6999955d189b
I20260808 18:31:13.272084 12562 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb: loading cluster ID for follower catalog manager: success
I20260808 18:31:13.274040 12562 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb: acquiring CA information for follower catalog manager: success
I20260808 18:31:13.275345 12562 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d01391cd853448a5bbb6627e9e8de3bb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 18:31:13.501888 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 12283
I20260808 18:31:13.511580 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 12365
I20260808 18:31:13.520009 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 12445
2026-08-08T18:31:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3341 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:31:13 dist-test-slave-gp6n krb5kdc[12596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:31:13 dist-test-slave-gp6n krb5kdc[12596](info): set up 2 sockets
Aug 08 18:31:13 dist-test-slave-gp6n krb5kdc[12596](info): commencing operation
krb5kdc: starting...
W20260808 18:31:14.634198 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:31:14 dist-test-slave-gp6n krb5kdc[12596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:31:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:31:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:31:14.765678 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:35119
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:35119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:31:14.880187 12609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:14.880489 12609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:14.880559 12609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:14.884609 12609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:31:14.884693 12609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:14.884735 12609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:14.884795 12609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:31:14.884835 12609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:31:14.889988 12609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:35119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:35119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:14.891065 12609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:14.892176 12609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:14.900928 12616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:14.901129 12609 server_base.cc:1034] running on GCE node
W20260808 18:31:14.900954 12615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:14.900940 12618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:14.901873 12609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:14.903043 12609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:14.904239 12609 hybrid_clock.cc:648] HybridClock initialized: now 1786213874904217 us; error 41 us; skew 500 ppm
Aug 08 18:31:14 dist-test-slave-gp6n krb5kdc[12596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213874, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:14.907135 12609 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:31:14.908038 12609 webserver.cc:460] Webserver started at http://127.29.78.190:44733/ using document root <none> and password file <none>
I20260808 18:31:14.908639 12609 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:14.908829 12609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:14.909089 12609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:31:14.911248 12609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "f78d3f835ef84af1a424c8026b829f5d"
format_stamp: "Formatted at 2026-08-08 18:31:14 on dist-test-slave-gp6n"
server_key: "d74e91e656e61b9648f43eae549f3fff"
server_key_iv: "1af14a6efc09334a4e42edccd237cfa5"
server_key_version: "encryptionkey@0"
I20260808 18:31:14.911897 12609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "f78d3f835ef84af1a424c8026b829f5d"
format_stamp: "Formatted at 2026-08-08 18:31:14 on dist-test-slave-gp6n"
server_key: "d74e91e656e61b9648f43eae549f3fff"
server_key_iv: "1af14a6efc09334a4e42edccd237cfa5"
server_key_version: "encryptionkey@0"
I20260808 18:31:14.915653 12609 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:31:14.918097 12627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:14.919371 12609 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:31:14.919518 12609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "f78d3f835ef84af1a424c8026b829f5d"
format_stamp: "Formatted at 2026-08-08 18:31:14 on dist-test-slave-gp6n"
server_key: "d74e91e656e61b9648f43eae549f3fff"
server_key_iv: "1af14a6efc09334a4e42edccd237cfa5"
server_key_version: "encryptionkey@0"
I20260808 18:31:14.919638 12609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:14.942768 12609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:14.945838 12609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:14.945994 12609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:14.953495 12609 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:35119
I20260808 18:31:14.953532 12685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:35119 every 8 connection(s)
I20260808 18:31:14.954487 12609 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:31:14.957233 12686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:31:14.962711 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 12609
I20260808 18:31:14.962816 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:31:14.962947 12686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d: Bootstrap starting.
I20260808 18:31:14.963197 30010 external_mini_cluster.cc:1428] Setting key fd64bbcc7ccc31bc62de14847eb515d5
I20260808 18:31:14.965680 12686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d: Neither blocks nor log segments found. Creating new log.
I20260808 18:31:14.966593 12686 log.cc:824] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:14.968544 12686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d: No bootstrap required, opened a new log
Aug 08 18:31:14 dist-test-slave-gp6n krb5kdc[12596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:31:14.973272 12686 raft_consensus.cc:348] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78d3f835ef84af1a424c8026b829f5d" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35119 } }
I20260808 18:31:14.973599 12686 raft_consensus.cc:374] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:31:14.973663 12686 raft_consensus.cc:729] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f78d3f835ef84af1a424c8026b829f5d, State: Initialized, Role: FOLLOWER
I20260808 18:31:14.974155 12686 consensus_queue.cc:260] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78d3f835ef84af1a424c8026b829f5d" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35119 } }
I20260808 18:31:14.974284 12686 raft_consensus.cc:388] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:31:14.974370 12686 raft_consensus.cc:482] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:31:14.974512 12686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:31:14.975387 12686 raft_consensus.cc:504] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78d3f835ef84af1a424c8026b829f5d" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35119 } }
I20260808 18:31:14.975739 12686 leader_election.cc:302] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f78d3f835ef84af1a424c8026b829f5d; no voters:
I20260808 18:31:14.976075 12686 leader_election.cc:288] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:31:14.976197 12694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:31:14.976444 12694 raft_consensus.cc:686] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d [term 1 LEADER]: Becoming Leader. State: Replica: f78d3f835ef84af1a424c8026b829f5d, State: Running, Role: LEADER
I20260808 18:31:14.976754 12694 consensus_queue.cc:237] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78d3f835ef84af1a424c8026b829f5d" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35119 } }
I20260808 18:31:14.977490 12686 sys_catalog.cc:564] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:31:14.978605 12695 sys_catalog.cc:455] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f78d3f835ef84af1a424c8026b829f5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78d3f835ef84af1a424c8026b829f5d" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35119 } } }
I20260808 18:31:14.978729 12695 sys_catalog.cc:458] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d [sys.catalog]: This master's current role is: LEADER
I20260808 18:31:14.978824 12690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:14.964558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42512 (local address 127.29.78.190:35119)
0808 18:31:14.965000 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:14.965019 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:14.965063 (+ 44us) server_negotiation.cc:407] Connection header received
0808 18:31:14.965962 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:14.966001 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:14.966442 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:14.966833 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:14.967603 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:14.968607 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:14.969845 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:14.970314 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:14.970500 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:14.973403 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:14.973409 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:14.973425 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:14.973457 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:14.975399 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:14.975859 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:14.975872 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:14.975876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:14.976029 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:14.976441 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:14.976444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:14.976446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:14.976949 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:14.977153 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:14.977429 (+ 276us) server_negotiation.cc:299] Negotiation successful
0808 18:31:14.977749 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":124,"threads_started":1}
I20260808 18:31:14.979003 12696 sys_catalog.cc:455] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f78d3f835ef84af1a424c8026b829f5d. Latest consensus state: current_term: 1 leader_uuid: "f78d3f835ef84af1a424c8026b829f5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f78d3f835ef84af1a424c8026b829f5d" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35119 } } }
I20260808 18:31:14.979079 12696 sys_catalog.cc:458] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d [sys.catalog]: This master's current role is: LEADER
I20260808 18:31:14.979349 12704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:31:14.981920 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:31:14.982455 12704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:31:14.987742 12704 catalog_manager.cc:1347] Generated new cluster ID: 89dbc23a7acd422fa7879adf78562a31
I20260808 18:31:14.987811 12704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:31:15.009114 12704 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:31:15.010123 12704 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:31:15.015395 12704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f78d3f835ef84af1a424c8026b829f5d: Generated new TSK 0
I20260808 18:31:15.016057 12704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:31:15.037999 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 12609
2026-08-08T18:31:15Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1517 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:31:15 dist-test-slave-gp6n krb5kdc[12721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:31:15 dist-test-slave-gp6n krb5kdc[12721](info): set up 2 sockets
Aug 08 18:31:15 dist-test-slave-gp6n krb5kdc[12721](info): commencing operation
krb5kdc: starting...
W20260808 18:31:16.135110 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:31:16 dist-test-slave-gp6n krb5kdc[12721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:31:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:31:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:31:16.269089 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:45639
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:45639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:31:16.379352 12736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:16.379654 12736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:16.379727 12736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:16.383749 12736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:31:16.383817 12736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:16.383854 12736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:16.383893 12736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:31:16.383929 12736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:31:16.388978 12736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:45639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:45639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:16.389987 12736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:16.390973 12736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:16.399266 12745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:16.399271 12743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:16.399437 12736 server_base.cc:1034] running on GCE node
W20260808 18:31:16.399335 12742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:16.400115 12736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:16.401099 12736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:16.402261 12736 hybrid_clock.cc:648] HybridClock initialized: now 1786213876402241 us; error 31 us; skew 500 ppm
Aug 08 18:31:16 dist-test-slave-gp6n krb5kdc[12721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213876, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:16.404913 12736 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:31:16.405814 12736 webserver.cc:460] Webserver started at http://127.29.78.190:41443/ using document root <none> and password file <none>
I20260808 18:31:16.406311 12736 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:16.406368 12736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:16.406597 12736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:31:16.408547 12736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "38f4095c21994a79abd088e4f600b669"
format_stamp: "Formatted at 2026-08-08 18:31:16 on dist-test-slave-gp6n"
server_key: "04084db741c8336a1733612f86f37336"
server_key_iv: "9bd326816e3a4ebafae7bcd58401684e"
server_key_version: "encryptionkey@0"
I20260808 18:31:16.409055 12736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "38f4095c21994a79abd088e4f600b669"
format_stamp: "Formatted at 2026-08-08 18:31:16 on dist-test-slave-gp6n"
server_key: "04084db741c8336a1733612f86f37336"
server_key_iv: "9bd326816e3a4ebafae7bcd58401684e"
server_key_version: "encryptionkey@0"
I20260808 18:31:16.412660 12736 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 18:31:16.414974 12753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:16.416191 12736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:31:16.416325 12736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "38f4095c21994a79abd088e4f600b669"
format_stamp: "Formatted at 2026-08-08 18:31:16 on dist-test-slave-gp6n"
server_key: "04084db741c8336a1733612f86f37336"
server_key_iv: "9bd326816e3a4ebafae7bcd58401684e"
server_key_version: "encryptionkey@0"
I20260808 18:31:16.416420 12736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:16.432926 12736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:16.435680 12736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:16.435875 12736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:16.443071 12736 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:45639
I20260808 18:31:16.443430 12812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:45639 every 8 connection(s)
I20260808 18:31:16.444082 12736 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:31:16.444991 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 12736
I20260808 18:31:16.445127 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:31:16.445421 30010 external_mini_cluster.cc:1428] Setting key 2e22679d6be219403d194b05acd9591c
I20260808 18:31:16.447258 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:31:16.453815 12813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669: Bootstrap starting.
Aug 08 18:31:16 dist-test-slave-gp6n krb5kdc[12721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:31:16.456282 12813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669: Neither blocks nor log segments found. Creating new log.
I20260808 18:31:16.457186 12813 log.cc:824] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:16.459152 12813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669: No bootstrap required, opened a new log
I20260808 18:31:16.460449 12816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:16.446891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43924 (local address 127.29.78.190:45639)
0808 18:31:16.447321 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:16.447351 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:16.447393 (+ 42us) server_negotiation.cc:407] Connection header received
0808 18:31:16.448207 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:16.448228 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:16.448644 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:16.449005 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:16.449690 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:16.451222 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:16.452090 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:16.452790 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:16.452951 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:16.455546 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:16.455553 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:16.455568 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:16.455598 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:16.457627 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:16.458093 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:16.458103 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:16.458107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:16.458234 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:16.458584 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:16.458591 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:16.458593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:16.458977 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:16.459194 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:16.459473 (+ 279us) server_negotiation.cc:299] Negotiation successful
0808 18:31:16.459760 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":117,"threads_started":1}
I20260808 18:31:16.463258 12813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f4095c21994a79abd088e4f600b669" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45639 } }
I20260808 18:31:16.463502 12813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:31:16.463591 12813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f4095c21994a79abd088e4f600b669, State: Initialized, Role: FOLLOWER
I20260808 18:31:16.464071 12813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f4095c21994a79abd088e4f600b669" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45639 } }
I20260808 18:31:16.464237 12813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:31:16.464305 12813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:31:16.464401 12813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:31:16.465286 12813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f4095c21994a79abd088e4f600b669" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45639 } }
I20260808 18:31:16.465575 12813 leader_election.cc:302] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38f4095c21994a79abd088e4f600b669; no voters:
I20260808 18:31:16.465834 12813 leader_election.cc:288] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:31:16.465993 12818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:31:16.466288 12818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 1 LEADER]: Becoming Leader. State: Replica: 38f4095c21994a79abd088e4f600b669, State: Running, Role: LEADER
I20260808 18:31:16.466639 12818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f4095c21994a79abd088e4f600b669" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45639 } }
I20260808 18:31:16.467078 12813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:31:16.468237 12820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38f4095c21994a79abd088e4f600b669. Latest consensus state: current_term: 1 leader_uuid: "38f4095c21994a79abd088e4f600b669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f4095c21994a79abd088e4f600b669" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45639 } } }
I20260808 18:31:16.468375 12820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [sys.catalog]: This master's current role is: LEADER
I20260808 18:31:16.468871 12828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:31:16.468878 12819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38f4095c21994a79abd088e4f600b669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f4095c21994a79abd088e4f600b669" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45639 } } }
I20260808 18:31:16.469521 12819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [sys.catalog]: This master's current role is: LEADER
I20260808 18:31:16.472050 12828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:31:16.477277 12828 catalog_manager.cc:1347] Generated new cluster ID: 699d79ad19fc4cc382279440dfc7bd53
I20260808 18:31:16.477351 12828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:31:16.518612 12828 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:31:16.519575 12828 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:31:16.525943 12828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669: Generated new TSK 0
I20260808 18:31:16.526705 12828 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260808 18:31:16.559031 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:45639
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:31:16.683197 12840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:16.683455 12840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:16.683498 12840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:16.687642 12840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:16.687727 12840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:16.687826 12840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:31:16.692991 12840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:45639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:16.693996 12840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:16.695053 12840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:16.704494 12848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:16.704532 12845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:16.704567 12846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:16.704665 12840 server_base.cc:1034] running on GCE node
I20260808 18:31:16.705163 12840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:16.705816 12840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:16.706997 12840 hybrid_clock.cc:648] HybridClock initialized: now 1786213876706984 us; error 24 us; skew 500 ppm
Aug 08 18:31:16 dist-test-slave-gp6n krb5kdc[12721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213876, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:16.709839 12840 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:31:16.710660 12840 webserver.cc:460] Webserver started at http://127.29.78.129:42479/ using document root <none> and password file <none>
I20260808 18:31:16.711149 12840 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:16.711205 12840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:16.711354 12840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:31:16.713238 12840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2"
format_stamp: "Formatted at 2026-08-08 18:31:16 on dist-test-slave-gp6n"
server_key: "c0b5a9370613bd4d6c0096a25fe9009c"
server_key_iv: "87cbbb51aac43b654ad015f3c5b23845"
server_key_version: "encryptionkey@0"
I20260808 18:31:16.713738 12840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2"
format_stamp: "Formatted at 2026-08-08 18:31:16 on dist-test-slave-gp6n"
server_key: "c0b5a9370613bd4d6c0096a25fe9009c"
server_key_iv: "87cbbb51aac43b654ad015f3c5b23845"
server_key_version: "encryptionkey@0"
I20260808 18:31:16.717265 12840 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 18:31:16.719599 12857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:16.720913 12840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:31:16.721021 12840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2"
format_stamp: "Formatted at 2026-08-08 18:31:16 on dist-test-slave-gp6n"
server_key: "c0b5a9370613bd4d6c0096a25fe9009c"
server_key_iv: "87cbbb51aac43b654ad015f3c5b23845"
server_key_version: "encryptionkey@0"
I20260808 18:31:16.721109 12840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:16.743245 12840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:16.746585 12840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:16.746737 12840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:16.747254 12840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:31:16.748018 12840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:31:16.748064 12840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:16.748104 12840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:31:16.748119 12840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:16.756875 12840 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:39627
I20260808 18:31:16.756903 12985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:39627 every 8 connection(s)
I20260808 18:31:16.757802 12840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:31:16.765990 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 12840
I20260808 18:31:16.766136 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:31:16.766511 30010 external_mini_cluster.cc:1428] Setting key ea9f831d2c399767462abc8875c32ab6
Aug 08 18:31:16 dist-test-slave-gp6n krb5kdc[12721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213876, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:31:16.772343 12816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:16.759677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:36335 (local address 127.29.78.190:45639)
0808 18:31:16.759816 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:16.759821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:16.760681 (+ 860us) server_negotiation.cc:407] Connection header received
0808 18:31:16.762035 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:16.762038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:16.762086 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:16.762163 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:16.763336 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:16.763891 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:16.764977 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:16.765215 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:16.765291 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:16.768295 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:16.768299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:16.768302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:16.768329 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:16.770098 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:16.770763 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:16.770766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:16.770767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:16.770844 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:16.771243 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:16.771245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:16.771246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:16.771411 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:16.771536 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:16.772121 (+ 585us) server_negotiation.cc:299] Negotiation successful
0808 18:31:16.772226 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 18:31:16.773542 12989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:16.760065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.129:36335)
0808 18:31:16.760524 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:16.760567 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:16.761832 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:16.762289 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:16.762295 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:16.762732 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:16.763196 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:16.763209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:16.764000 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:16.764003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:16.764871 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:16.764879 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:16.765345 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:16.765348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:16.765505 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:16.766148 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:16.766171 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:16.768125 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:16.770220 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:16.770224 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:16.770235 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:16.770664 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:16.770938 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:16.770944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:16.770947 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:16.771135 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:16.771521 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:16.771530 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:16.771861 (+ 331us) client_negotiation.cc:769] Sending connection context
0808 18:31:16.772061 (+ 200us) client_negotiation.cc:240] Negotiation successful
0808 18:31:16.772412 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":118,"threads_started":1}
I20260808 18:31:16.774888 12986 heartbeater.cc:347] Connected to a master server at 127.29.78.190:45639
I20260808 18:31:16.775238 12986 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:16.775722 12986 heartbeater.cc:511] Master 127.29.78.190:45639 requested a full tablet report, sending...
I20260808 18:31:16.777187 12773 ts_manager.cc:194] Registered new tserver with Master: 7d40e86dd6fe451aad3a8c90a4bef3c2 (127.29.78.129:39627)
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260808 18:31:16.778352 12773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:36335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260808 18:31:16.798296 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:45639
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:31:16.910357 12992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:16.910640 12992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:16.910717 12992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:16.914659 12992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:16.914731 12992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:16.914816 12992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:31:16.920183 12992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:45639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:16.921303 12992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:16.922319 12992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:16.931900 12999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:16.932042 12992 server_base.cc:1034] running on GCE node
W20260808 18:31:16.932137 13003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:16.931922 12997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:16.932662 12992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:16.933351 12992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:16.934549 12992 hybrid_clock.cc:648] HybridClock initialized: now 1786213876934514 us; error 47 us; skew 500 ppm
Aug 08 18:31:16 dist-test-slave-gp6n krb5kdc[12721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213876, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:16.942905 12992 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:31:16.943858 12992 webserver.cc:460] Webserver started at http://127.29.78.130:40395/ using document root <none> and password file <none>
I20260808 18:31:16.944500 12992 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:16.944591 12992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:16.944828 12992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:31:16.946951 12992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "09ca2a2882524822841a90e33b0d8ad8"
format_stamp: "Formatted at 2026-08-08 18:31:16 on dist-test-slave-gp6n"
server_key: "6628d4e8c4e24dff6a637a7db5055565"
server_key_iv: "c1666259adfc9eb05bdd79e2c7809df9"
server_key_version: "encryptionkey@0"
I20260808 18:31:16.947538 12992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "09ca2a2882524822841a90e33b0d8ad8"
format_stamp: "Formatted at 2026-08-08 18:31:16 on dist-test-slave-gp6n"
server_key: "6628d4e8c4e24dff6a637a7db5055565"
server_key_iv: "c1666259adfc9eb05bdd79e2c7809df9"
server_key_version: "encryptionkey@0"
I20260808 18:31:16.951295 12992 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:31:16.953579 13012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:16.954864 12992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:31:16.954996 12992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "09ca2a2882524822841a90e33b0d8ad8"
format_stamp: "Formatted at 2026-08-08 18:31:16 on dist-test-slave-gp6n"
server_key: "6628d4e8c4e24dff6a637a7db5055565"
server_key_iv: "c1666259adfc9eb05bdd79e2c7809df9"
server_key_version: "encryptionkey@0"
I20260808 18:31:16.955103 12992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:16.973398 12992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:16.976295 12992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:16.976527 12992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:16.977114 12992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:31:16.977927 12992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:31:16.977998 12992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:16.978070 12992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:31:16.978121 12992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:16.987011 12992 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:38867
I20260808 18:31:16.987041 13138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:38867 every 8 connection(s)
I20260808 18:31:16.987866 12992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:31:16.994971 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 12992
I20260808 18:31:16.995086 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:31:16.995379 30010 external_mini_cluster.cc:1428] Setting key 4c02fec2eec867d5404950579f2f7f4f
Aug 08 18:31:16 dist-test-slave-gp6n krb5kdc[12721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213876, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:31:17.001797 12816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:16.989714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:58165 (local address 127.29.78.190:45639)
0808 18:31:16.989846 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:16.989850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:16.990658 (+ 808us) server_negotiation.cc:407] Connection header received
0808 18:31:16.992053 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:16.992056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:16.992113 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:16.992223 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:16.993355 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:16.993894 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:16.995284 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:16.995562 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:16.995651 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:16.998230 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:16.998233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:16.998236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:16.998266 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:16.999751 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:17.000279 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:17.000284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:17.000286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:17.000370 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:17.000715 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:17.000720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:17.000721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:17.000898 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:17.001031 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:17.001515 (+ 484us) server_negotiation.cc:299] Negotiation successful
0808 18:31:17.001639 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 18:31:17.002687 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:16.990019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.130:58165)
0808 18:31:16.990515 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:16.990557 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:16.991836 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:16.992402 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:16.992409 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:16.992752 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:16.993212 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:16.993224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:16.994023 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:16.994026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:16.995160 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:16.995171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:16.995717 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:16.995722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:16.995900 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:16.996511 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:16.996531 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:16.998070 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:16.999881 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:16.999886 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:16.999899 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:17.000173 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:17.000464 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:17.000468 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:17.000470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:17.000617 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:17.001043 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:17.001050 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:17.001277 (+ 227us) client_negotiation.cc:769] Sending connection context
0808 18:31:17.001468 (+ 191us) client_negotiation.cc:240] Negotiation successful
0808 18:31:17.001799 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":112,"threads_started":1}
I20260808 18:31:17.003854 13139 heartbeater.cc:347] Connected to a master server at 127.29.78.190:45639
I20260808 18:31:17.004104 13139 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:17.004591 13139 heartbeater.cc:511] Master 127.29.78.190:45639 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260808 18:31:17.005679 12773 ts_manager.cc:194] Registered new tserver with Master: 09ca2a2882524822841a90e33b0d8ad8 (127.29.78.130:38867)
I20260808 18:31:17.006294 12773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:58165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260808 18:31:17.026037 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:45639
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:31:17.138953 13144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:17.139191 13144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:17.139256 13144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:17.143222 13144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:17.143275 13144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:17.143376 13144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:31:17.148754 13144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:45639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:17.149775 13144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:17.150868 13144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:17.160526 13151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:17.160635 13153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:17.160531 13150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:17.160765 13144 server_base.cc:1034] running on GCE node
I20260808 18:31:17.161264 13144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:17.162006 13144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:17.163177 13144 hybrid_clock.cc:648] HybridClock initialized: now 1786213877163157 us; error 30 us; skew 500 ppm
Aug 08 18:31:17 dist-test-slave-gp6n krb5kdc[12721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213877, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:17.165990 13144 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:31:17.166930 13144 webserver.cc:460] Webserver started at http://127.29.78.131:38181/ using document root <none> and password file <none>
I20260808 18:31:17.167541 13144 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:17.167618 13144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:17.167840 13144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:31:17.169893 13144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "801408a989244177ae8b1bd0a14daa61"
format_stamp: "Formatted at 2026-08-08 18:31:17 on dist-test-slave-gp6n"
server_key: "3bfce5bb4a85eb52eedd5e693feba061"
server_key_iv: "2baaa849bca8e70f93daa7a63f3726e9"
server_key_version: "encryptionkey@0"
I20260808 18:31:17.170459 13144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "801408a989244177ae8b1bd0a14daa61"
format_stamp: "Formatted at 2026-08-08 18:31:17 on dist-test-slave-gp6n"
server_key: "3bfce5bb4a85eb52eedd5e693feba061"
server_key_iv: "2baaa849bca8e70f93daa7a63f3726e9"
server_key_version: "encryptionkey@0"
I20260808 18:31:17.174042 13144 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:31:17.176359 13160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:17.177552 13144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:31:17.177672 13144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "801408a989244177ae8b1bd0a14daa61"
format_stamp: "Formatted at 2026-08-08 18:31:17 on dist-test-slave-gp6n"
server_key: "3bfce5bb4a85eb52eedd5e693feba061"
server_key_iv: "2baaa849bca8e70f93daa7a63f3726e9"
server_key_version: "encryptionkey@0"
I20260808 18:31:17.177773 13144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:17.198372 13144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:17.201440 13144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:17.201638 13144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:17.202229 13144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:31:17.203084 13144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:31:17.203132 13144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:17.203199 13144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:31:17.203231 13144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:17.212215 13144 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:36891
I20260808 18:31:17.212239 13285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:36891 every 8 connection(s)
I20260808 18:31:17.213114 13144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
Aug 08 18:31:17 dist-test-slave-gp6n krb5kdc[12721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213877, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:31:17.222987 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 13144
I20260808 18:31:17.223074 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:31:17.223367 30010 external_mini_cluster.cc:1428] Setting key 11d6cf9160afc178c4f7744315c18a4b
I20260808 18:31:17.227092 12816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:17.214969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:46405 (local address 127.29.78.190:45639)
0808 18:31:17.215124 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:17.215128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:17.215933 (+ 805us) server_negotiation.cc:407] Connection header received
0808 18:31:17.217330 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:17.217333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:17.217380 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:17.217458 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:17.218644 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.219194 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.220428 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.220665 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.220746 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:17.223293 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:17.223296 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:17.223298 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:17.223321 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:17.225219 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:17.225777 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:17.225780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:17.225781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:17.225854 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:17.226163 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:17.226165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:17.226166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:17.226330 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:17.226478 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:17.226879 (+ 401us) server_negotiation.cc:299] Negotiation successful
0808 18:31:17.226978 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 18:31:17.227883 13288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:17.215307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.131:46405)
0808 18:31:17.215768 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:17.215810 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:17.217132 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:17.217635 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:17.217641 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:17.217992 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:17.218514 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:17.218530 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.219353 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.219356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:17.220308 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:17.220317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.220794 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.220797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:17.220940 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:17.221561 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:17.221576 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:17.223078 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:17.225329 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:17.225333 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:17.225343 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:17.225678 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:17.225939 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:17.225942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:17.225943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:17.226078 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:17.226468 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:17.226473 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:17.226681 (+ 208us) client_negotiation.cc:769] Sending connection context
0808 18:31:17.226830 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 18:31:17.227071 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":119,"threads_started":1}
I20260808 18:31:17.228924 13286 heartbeater.cc:347] Connected to a master server at 127.29.78.190:45639
I20260808 18:31:17.229151 13286 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:17.229624 13286 heartbeater.cc:511] Master 127.29.78.190:45639 requested a full tablet report, sending...
I20260808 18:31:17.230597 12773 ts_manager.cc:194] Registered new tserver with Master: 801408a989244177ae8b1bd0a14daa61 (127.29.78.131:36891)
I20260808 18:31:17.231144 12773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:46405
I20260808 18:31:17.237249 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:31:17.246276 12816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:17.239282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43938 (local address 127.29.78.190:45639)
0808 18:31:17.239400 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:17.239404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:17.239626 (+ 222us) server_negotiation.cc:407] Connection header received
0808 18:31:17.239709 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:17.239711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:17.239751 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:17.239819 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:17.240468 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.240986 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.241805 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.242021 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.242104 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:17.243419 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:17.243422 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:17.243424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:17.243444 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:17.244890 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:17.245286 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:17.245288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:17.245290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:17.245359 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:17.245582 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:17.245584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:17.245585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:17.245774 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:17.245875 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:17.246080 (+ 205us) server_negotiation.cc:299] Negotiation successful
0808 18:31:17.246167 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 18:31:17.249045 12773 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:31:17.252027 12773 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:31:17.266296 13299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:17.260220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:39627 (local address 127.0.0.1:49754)
0808 18:31:17.260929 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:17.260947 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:17.261035 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:17.262074 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:17.262078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:17.262096 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:17.262187 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:17.262192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.263266 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.263269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:17.264759 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:17.264771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.265915 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.265918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:17.266001 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:17.266025 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 18:31:17.266101 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 18:31:17.266156 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":16,"thread_start_us":101,"threads_started":1}
I20260808 18:31:17.266562 13300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:17.260666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36888 (local address 127.29.78.131:36891)
0808 18:31:17.261064 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:17.261078 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:17.261127 (+ 49us) server_negotiation.cc:407] Connection header received
0808 18:31:17.261210 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:17.261215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:17.261426 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:17.261932 (+ 506us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:17.262096 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.263324 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.264895 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.265496 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.265623 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:17.265767 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:17.266276 (+ 509us) server_negotiation.cc:299] Negotiation successful
0808 18:31:17.266378 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":191,"threads_started":1}
I20260808 18:31:17.266201 13297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:17.260017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:36891 (local address 127.0.0.1:36888)
0808 18:31:17.260492 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:17.260540 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:17.260665 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:17.261607 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:17.261614 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:17.261640 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:17.261780 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:17.261786 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.263452 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.263456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:17.264779 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:17.264785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.265623 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.265627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:17.265732 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:17.265767 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 18:31:17.265905 (+ 138us) client_negotiation.cc:240] Negotiation successful
0808 18:31:17.266041 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":85,"thread_start_us":71,"threads_started":1}
I20260808 18:31:17.266793 13302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:17.261244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49754 (local address 127.29.78.129:39627)
0808 18:31:17.261622 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:17.261639 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:17.261661 (+ 22us) server_negotiation.cc:407] Connection header received
0808 18:31:17.261735 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:17.261740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:17.261911 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:17.262057 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:17.262301 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.263113 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.264924 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.265740 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.265883 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:17.265999 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:17.266150 (+ 151us) server_negotiation.cc:299] Negotiation successful
0808 18:31:17.266245 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":111,"threads_started":1}
I20260808 18:31:17.266916 13301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:17.260813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51224 (local address 127.29.78.130:38867)
0808 18:31:17.261276 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:17.261292 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:17.261534 (+ 242us) server_negotiation.cc:407] Connection header received
0808 18:31:17.261741 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:17.261745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:17.261911 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:17.262041 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:17.262505 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.263572 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.264929 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.265740 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.265858 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:17.265999 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:17.266501 (+ 502us) server_negotiation.cc:299] Negotiation successful
0808 18:31:17.266595 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":147,"threads_started":1}
I20260808 18:31:17.267230 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:17.260417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:38867 (local address 127.0.0.1:51224)
0808 18:31:17.261431 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:17.261444 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:17.261602 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:17.262240 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:17.262264 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:17.262286 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:17.262374 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:17.262380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.263693 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.263699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:17.264805 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:17.264814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.266091 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.266096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:17.266187 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:17.266209 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 18:31:17.266280 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 18:31:17.267109 (+ 829us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"mutex_wait_us":35,"thread_start_us":117,"threads_started":1}
I20260808 18:31:17.269621 13213 tablet_service.cc:1467] Processing CreateTablet for tablet 43222644e66048e28f929dcecd716be8 (DEFAULT_TABLE table=test-table [id=7993d379a8044f73b3f092d2b4d0e33e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:31:17.269543 12917 tablet_service.cc:1467] Processing CreateTablet for tablet 43222644e66048e28f929dcecd716be8 (DEFAULT_TABLE table=test-table [id=7993d379a8044f73b3f092d2b4d0e33e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:31:17.269946 13067 tablet_service.cc:1467] Processing CreateTablet for tablet 43222644e66048e28f929dcecd716be8 (DEFAULT_TABLE table=test-table [id=7993d379a8044f73b3f092d2b4d0e33e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:31:17.270646 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43222644e66048e28f929dcecd716be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:31:17.270645 12917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43222644e66048e28f929dcecd716be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:31:17.270838 13067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43222644e66048e28f929dcecd716be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:31:17.277237 13303 tablet_bootstrap.cc:492] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61: Bootstrap starting.
I20260808 18:31:17.278529 13306 tablet_bootstrap.cc:492] T 43222644e66048e28f929dcecd716be8 P 09ca2a2882524822841a90e33b0d8ad8: Bootstrap starting.
I20260808 18:31:17.278765 13304 tablet_bootstrap.cc:492] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2: Bootstrap starting.
I20260808 18:31:17.279780 13303 tablet_bootstrap.cc:654] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61: Neither blocks nor log segments found. Creating new log.
I20260808 18:31:17.280505 13303 log.cc:824] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:17.280617 13304 tablet_bootstrap.cc:654] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2: Neither blocks nor log segments found. Creating new log.
I20260808 18:31:17.281077 13306 tablet_bootstrap.cc:654] T 43222644e66048e28f929dcecd716be8 P 09ca2a2882524822841a90e33b0d8ad8: Neither blocks nor log segments found. Creating new log.
I20260808 18:31:17.281543 13304 log.cc:824] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:17.281919 13306 log.cc:824] T 43222644e66048e28f929dcecd716be8 P 09ca2a2882524822841a90e33b0d8ad8: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:17.282603 13303 tablet_bootstrap.cc:492] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61: No bootstrap required, opened a new log
I20260808 18:31:17.282801 13303 ts_tablet_manager.cc:1397] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:31:17.283782 13304 tablet_bootstrap.cc:492] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2: No bootstrap required, opened a new log
I20260808 18:31:17.283990 13304 ts_tablet_manager.cc:1397] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 18:31:17.284053 13306 tablet_bootstrap.cc:492] T 43222644e66048e28f929dcecd716be8 P 09ca2a2882524822841a90e33b0d8ad8: No bootstrap required, opened a new log
I20260808 18:31:17.284219 13306 ts_tablet_manager.cc:1397] T 43222644e66048e28f929dcecd716be8 P 09ca2a2882524822841a90e33b0d8ad8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:31:17.286657 13303 raft_consensus.cc:348] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } }
I20260808 18:31:17.286927 13303 raft_consensus.cc:374] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:31:17.286994 13303 raft_consensus.cc:729] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 801408a989244177ae8b1bd0a14daa61, State: Initialized, Role: FOLLOWER
I20260808 18:31:17.287370 13304 raft_consensus.cc:348] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } }
I20260808 18:31:17.287448 13303 consensus_queue.cc:260] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } }
I20260808 18:31:17.287582 13304 raft_consensus.cc:374] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:31:17.287643 13304 raft_consensus.cc:729] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d40e86dd6fe451aad3a8c90a4bef3c2, State: Initialized, Role: FOLLOWER
I20260808 18:31:17.288034 13304 consensus_queue.cc:260] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } }
I20260808 18:31:17.287949 13306 raft_consensus.cc:348] T 43222644e66048e28f929dcecd716be8 P 09ca2a2882524822841a90e33b0d8ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } }
I20260808 18:31:17.288167 13306 raft_consensus.cc:374] T 43222644e66048e28f929dcecd716be8 P 09ca2a2882524822841a90e33b0d8ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:31:17.288232 13306 raft_consensus.cc:729] T 43222644e66048e28f929dcecd716be8 P 09ca2a2882524822841a90e33b0d8ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ca2a2882524822841a90e33b0d8ad8, State: Initialized, Role: FOLLOWER
I20260808 18:31:17.288439 13303 ts_tablet_manager.cc:1428] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:31:17.288702 12986 heartbeater.cc:503] Master 127.29.78.190:45639 was elected leader, sending a full tablet report...
I20260808 18:31:17.288749 13286 heartbeater.cc:503] Master 127.29.78.190:45639 was elected leader, sending a full tablet report...
I20260808 18:31:17.288935 13304 ts_tablet_manager.cc:1428] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 18:31:17.289381 13306 consensus_queue.cc:260] T 43222644e66048e28f929dcecd716be8 P 09ca2a2882524822841a90e33b0d8ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } }
I20260808 18:31:17.290268 13306 ts_tablet_manager.cc:1428] T 43222644e66048e28f929dcecd716be8 P 09ca2a2882524822841a90e33b0d8ad8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:31:17.290611 13139 heartbeater.cc:503] Master 127.29.78.190:45639 was elected leader, sending a full tablet report...
I20260808 18:31:17.309396 13311 raft_consensus.cc:482] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:31:17.309535 13311 raft_consensus.cc:504] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } }
I20260808 18:31:17.310537 13311 leader_election.cc:288] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d40e86dd6fe451aad3a8c90a4bef3c2 (127.29.78.129:39627), 09ca2a2882524822841a90e33b0d8ad8 (127.29.78.130:38867)
I20260808 18:31:17.314180 13288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:17.310665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:39627 (local address 127.29.78.131:33353)
0808 18:31:17.310801 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:17.310817 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:17.310927 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:17.311246 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:17.311250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:17.311285 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:17.311408 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:17.311418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.312249 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.312252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:17.313091 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:17.313100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.313741 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.313748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:17.313860 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:17.313884 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 18:31:17.313949 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 18:31:17.314032 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":1}
I20260808 18:31:17.314185 13302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:17.310867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:33353 (local address 127.29.78.129:39627)
0808 18:31:17.311001 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:17.311007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:17.311022 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:31:17.311068 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:17.311071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:17.311118 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:17.311223 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:17.311514 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.312145 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.313217 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.313625 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.313715 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:17.313791 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:17.313998 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 18:31:17.314067 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 18:31:17.314208 13301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:17.310882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:57305 (local address 127.29.78.130:38867)
0808 18:31:17.311010 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:17.311016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:17.311225 (+ 209us) server_negotiation.cc:407] Connection header received
0808 18:31:17.311329 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:17.311332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:17.311388 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:17.311468 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:17.311767 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.312260 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.313269 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.313654 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.313785 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:17.313835 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:17.314041 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 18:31:17.314109 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:31:17.314210 13314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:17.310770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:38867 (local address 127.29.78.131:57305)
0808 18:31:17.311120 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:17.311133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:17.311237 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:17.311509 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:17.311512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:17.311529 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:17.311664 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:17.311670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.312364 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.312367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:17.313118 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:17.313127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.313795 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.313799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:17.313896 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:17.313913 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 18:31:17.314009 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 18:31:17.314070 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":2,"thread_start_us":103,"threads_started":1}
I20260808 18:31:17.314787 13088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43222644e66048e28f929dcecd716be8" candidate_uuid: "801408a989244177ae8b1bd0a14daa61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ca2a2882524822841a90e33b0d8ad8" is_pre_election: true
I20260808 18:31:17.314889 12938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43222644e66048e28f929dcecd716be8" candidate_uuid: "801408a989244177ae8b1bd0a14daa61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" is_pre_election: true
I20260808 18:31:17.315076 13088 raft_consensus.cc:2454] T 43222644e66048e28f929dcecd716be8 P 09ca2a2882524822841a90e33b0d8ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 801408a989244177ae8b1bd0a14daa61 in term 0.
I20260808 18:31:17.315163 12938 raft_consensus.cc:2454] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 801408a989244177ae8b1bd0a14daa61 in term 0.
I20260808 18:31:17.315560 13165 leader_election.cc:302] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ca2a2882524822841a90e33b0d8ad8, 801408a989244177ae8b1bd0a14daa61; no voters:
I20260808 18:31:17.315829 13311 raft_consensus.cc:2781] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:31:17.315932 13311 raft_consensus.cc:482] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:31:17.315995 13311 raft_consensus.cc:3037] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:31:17.316975 13311 raft_consensus.cc:504] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } }
I20260808 18:31:17.317296 13311 leader_election.cc:288] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61 [CANDIDATE]: Term 1 election: Requested vote from peers 7d40e86dd6fe451aad3a8c90a4bef3c2 (127.29.78.129:39627), 09ca2a2882524822841a90e33b0d8ad8 (127.29.78.130:38867)
I20260808 18:31:17.317663 13088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43222644e66048e28f929dcecd716be8" candidate_uuid: "801408a989244177ae8b1bd0a14daa61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ca2a2882524822841a90e33b0d8ad8"
I20260808 18:31:17.317665 12938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43222644e66048e28f929dcecd716be8" candidate_uuid: "801408a989244177ae8b1bd0a14daa61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2"
I20260808 18:31:17.317756 13088 raft_consensus.cc:3037] T 43222644e66048e28f929dcecd716be8 P 09ca2a2882524822841a90e33b0d8ad8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:31:17.317811 12938 raft_consensus.cc:3037] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:31:17.318846 13088 raft_consensus.cc:2454] T 43222644e66048e28f929dcecd716be8 P 09ca2a2882524822841a90e33b0d8ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 801408a989244177ae8b1bd0a14daa61 in term 1.
I20260808 18:31:17.318948 12938 raft_consensus.cc:2454] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 801408a989244177ae8b1bd0a14daa61 in term 1.
I20260808 18:31:17.319226 13165 leader_election.cc:302] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09ca2a2882524822841a90e33b0d8ad8, 7d40e86dd6fe451aad3a8c90a4bef3c2, 801408a989244177ae8b1bd0a14daa61; no voters:
I20260808 18:31:17.319427 13311 raft_consensus.cc:2781] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:31:17.319686 13311 raft_consensus.cc:686] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61 [term 1 LEADER]: Becoming Leader. State: Replica: 801408a989244177ae8b1bd0a14daa61, State: Running, Role: LEADER
I20260808 18:31:17.320016 13311 consensus_queue.cc:237] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } }
I20260808 18:31:17.323014 12773 catalog_manager.cc:5495] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 801408a989244177ae8b1bd0a14daa61 (127.29.78.131). New cstate: current_term: 1 leader_uuid: "801408a989244177ae8b1bd0a14daa61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } health_report { overall_health: UNKNOWN } } }
I20260808 18:31:17.343688 13300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:17.340158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36902 (local address 127.29.78.131:36891)
0808 18:31:17.340296 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:17.340300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:17.340314 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:31:17.340502 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:17.340504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:17.340557 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:17.340646 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:31:17.340876 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.341368 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.342208 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.342457 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.342543 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:17.342819 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:31:17.343236 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:31:17.343325 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:17.343453 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 18:31:17.343504 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 18:31:17.349784 13088 raft_consensus.cc:1261] T 43222644e66048e28f929dcecd716be8 P 09ca2a2882524822841a90e33b0d8ad8 [term 1 FOLLOWER]: Refusing update from remote peer 801408a989244177ae8b1bd0a14daa61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:31:17.349784 12938 raft_consensus.cc:1261] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Refusing update from remote peer 801408a989244177ae8b1bd0a14daa61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:31:17.350548 13315 consensus_queue.cc:1035] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:31:17.350723 13311 consensus_queue.cc:1035] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:31:17.366612 12773 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:31:17.366804 12773 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:31:17.369244 12773 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669: Sending DeleteTablet for 3 replicas of tablet 43222644e66048e28f929dcecd716be8
I20260808 18:31:17.369956 13213 tablet_service.cc:1514] Processing DeleteTablet for tablet 43222644e66048e28f929dcecd716be8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:31:17 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:36888
I20260808 18:31:17.370074 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 12840
I20260808 18:31:17.370143 13067 tablet_service.cc:1514] Processing DeleteTablet for tablet 43222644e66048e28f929dcecd716be8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:31:17 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:51224
I20260808 18:31:17.371361 13330 tablet_replica.cc:330] T 43222644e66048e28f929dcecd716be8 P 09ca2a2882524822841a90e33b0d8ad8: stopping tablet replica
I20260808 18:31:17.371539 13329 tablet_replica.cc:330] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61: stopping tablet replica
I20260808 18:31:17.371757 13330 raft_consensus.cc:2229] T 43222644e66048e28f929dcecd716be8 P 09ca2a2882524822841a90e33b0d8ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:31:17.371918 13329 raft_consensus.cc:2229] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:31:17.372134 13330 raft_consensus.cc:2258] T 43222644e66048e28f929dcecd716be8 P 09ca2a2882524822841a90e33b0d8ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:31:17.372387 13329 raft_consensus.cc:2258] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:31:17.374056 13330 ts_tablet_manager.cc:1905] T 43222644e66048e28f929dcecd716be8 P 09ca2a2882524822841a90e33b0d8ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:31:17.374361 13329 ts_tablet_manager.cc:1905] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:31:17.377168 13329 ts_tablet_manager.cc:1918] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:31:17.377326 13329 log.cc:1196] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/43222644e66048e28f929dcecd716be8
I20260808 18:31:17.377694 13330 ts_tablet_manager.cc:1918] T 43222644e66048e28f929dcecd716be8 P 09ca2a2882524822841a90e33b0d8ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:31:17.377799 13330 log.cc:1196] T 43222644e66048e28f929dcecd716be8 P 09ca2a2882524822841a90e33b0d8ad8: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/43222644e66048e28f929dcecd716be8
I20260808 18:31:17.377739 13329 ts_tablet_manager.cc:1939] T 43222644e66048e28f929dcecd716be8 P 801408a989244177ae8b1bd0a14daa61: Deleting consensus metadata
I20260808 18:31:17.378149 13330 ts_tablet_manager.cc:1939] T 43222644e66048e28f929dcecd716be8 P 09ca2a2882524822841a90e33b0d8ad8: Deleting consensus metadata
I20260808 18:31:17.379140 12757 catalog_manager.cc:4841] TS 801408a989244177ae8b1bd0a14daa61 (127.29.78.131:36891): tablet 43222644e66048e28f929dcecd716be8 (table test-table [id=7993d379a8044f73b3f092d2b4d0e33e]) successfully deleted
I20260808 18:31:17.379339 12756 catalog_manager.cc:4841] TS 09ca2a2882524822841a90e33b0d8ad8 (127.29.78.130:38867): tablet 43222644e66048e28f929dcecd716be8 (table test-table [id=7993d379a8044f73b3f092d2b4d0e33e]) successfully deleted
W20260808 18:31:17.383059 12754 connection.cc:535] client connection to 127.29.78.129:39627 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 18:31:17.383162 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 12992
W20260808 18:31:17.383234 12754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 18:31:17.385856 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:17.384968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:39627 (local address 127.0.0.1:49758)
0808 18:31:17.385691 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:17.385776 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":601}
W20260808 18:31:17.386448 12754 catalog_manager.cc:4584] TS 7d40e86dd6fe451aad3a8c90a4bef3c2 (127.29.78.129:39627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 43222644e66048e28f929dcecd716be8: Network error: Client connection negotiation failed: client connection to 127.29.78.129:39627: connect: Connection refused (error 111)
I20260808 18:31:17.391414 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 13144
I20260808 18:31:17.398731 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 12736
I20260808 18:31:17.406893 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:45639
--webserver_interface=127.29.78.190
--webserver_port=41443
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:45639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:31:17.518239 13333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:17.518522 13333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:17.518661 13333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:17.522637 13333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:31:17.522693 13333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:17.522712 13333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:17.522760 13333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:31:17.522795 13333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:31:17.527866 13333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:45639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:45639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=41443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:17.528844 13333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:17.529851 13333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:17.544878 13343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:17.544858 13341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:17.544919 13333 server_base.cc:1034] running on GCE node
W20260808 18:31:17.544858 13340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:17.545653 13333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:17.546891 13333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:17.548058 13333 hybrid_clock.cc:648] HybridClock initialized: now 1786213877548041 us; error 27 us; skew 500 ppm
Aug 08 18:31:17 dist-test-slave-gp6n krb5kdc[12721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213877, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:17.550887 13333 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:31:17.551762 13333 webserver.cc:460] Webserver started at http://127.29.78.190:41443/ using document root <none> and password file <none>
I20260808 18:31:17.552343 13333 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:17.552441 13333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:17.556430 13333 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:31:17.558523 13352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:17.559748 13333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:31:17.559896 13333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "38f4095c21994a79abd088e4f600b669"
format_stamp: "Formatted at 2026-08-08 18:31:16 on dist-test-slave-gp6n"
server_key: "04084db741c8336a1733612f86f37336"
server_key_iv: "9bd326816e3a4ebafae7bcd58401684e"
server_key_version: "encryptionkey@0"
I20260808 18:31:17.560514 13333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:17.577107 13333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:17.579939 13333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:17.580143 13333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:17.587399 13333 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:45639
I20260808 18:31:17.587689 13414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:45639 every 8 connection(s)
I20260808 18:31:17.588541 13333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:31:17.592947 13416 sys_catalog.cc:263] Verifying existing consensus state
I20260808 18:31:17.593355 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 13333
I20260808 18:31:17.593837 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:39627
--local_ip_for_outbound_sockets=127.29.78.129
--tserver_master_addrs=127.29.78.190:45639
--webserver_port=42479
--webserver_interface=127.29.78.129
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:31:17.595196 13416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669: Bootstrap starting.
I20260808 18:31:17.612804 13416 log.cc:824] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:17.621490 13416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:31:17.621871 13416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669: Bootstrap complete.
I20260808 18:31:17.625897 13416 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f4095c21994a79abd088e4f600b669" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45639 } }
I20260808 18:31:17.626267 13416 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f4095c21994a79abd088e4f600b669, State: Initialized, Role: FOLLOWER
I20260808 18:31:17.626803 13416 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f4095c21994a79abd088e4f600b669" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45639 } }
I20260808 18:31:17.626948 13416 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:31:17.627020 13416 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:31:17.627136 13416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 1 FOLLOWER]: Advancing to term 2
I20260808 18:31:17.629045 13416 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f4095c21994a79abd088e4f600b669" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45639 } }
I20260808 18:31:17.629380 13416 leader_election.cc:302] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38f4095c21994a79abd088e4f600b669; no voters:
I20260808 18:31:17.629719 13416 leader_election.cc:288] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 18:31:17.629801 13421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 18:31:17.629987 13421 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 2 LEADER]: Becoming Leader. State: Replica: 38f4095c21994a79abd088e4f600b669, State: Running, Role: LEADER
I20260808 18:31:17.630261 13421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f4095c21994a79abd088e4f600b669" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45639 } }
I20260808 18:31:17.630949 13416 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:31:17.631198 13421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38f4095c21994a79abd088e4f600b669. Latest consensus state: current_term: 2 leader_uuid: "38f4095c21994a79abd088e4f600b669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f4095c21994a79abd088e4f600b669" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45639 } } }
I20260808 18:31:17.631316 13421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [sys.catalog]: This master's current role is: LEADER
I20260808 18:31:17.631940 13422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "38f4095c21994a79abd088e4f600b669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f4095c21994a79abd088e4f600b669" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45639 } } }
I20260808 18:31:17.632161 13425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:31:17.632196 13422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [sys.catalog]: This master's current role is: LEADER
I20260808 18:31:17.636802 13425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:31:17.637401 13425 catalog_manager.cc:1259] Loaded cluster ID: 699d79ad19fc4cc382279440dfc7bd53
I20260808 18:31:17.637458 13425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:31:17.638906 13425 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260808 18:31:17.643841 13425 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260808 18:31:17.649116 13425 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:31:17.649852 13425 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669: Loaded TSK: 0
I20260808 18:31:17.651041 13425 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 18:31:17.708761 13419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:17.709040 13419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:17.709112 13419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:17.713136 13419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:17.713218 13419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:17.713304 13419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:31:17.718290 13419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:39627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=42479
--tserver_master_addrs=127.29.78.190:45639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:17.719298 13419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:17.720335 13419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:17.730700 13446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:17.730713 13449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:17.730774 13447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:17.731220 13419 server_base.cc:1034] running on GCE node
I20260808 18:31:17.731616 13419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:17.732363 13419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:17.733537 13419 hybrid_clock.cc:648] HybridClock initialized: now 1786213877733515 us; error 37 us; skew 500 ppm
Aug 08 18:31:17 dist-test-slave-gp6n krb5kdc[12721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213877, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:17.736339 13419 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:31:17.737287 13419 webserver.cc:460] Webserver started at http://127.29.78.129:42479/ using document root <none> and password file <none>
I20260808 18:31:17.737886 13419 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:17.737972 13419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:17.741999 13419 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:31:17.744156 13457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:17.745417 13419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 18:31:17.745512 13419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2"
format_stamp: "Formatted at 2026-08-08 18:31:16 on dist-test-slave-gp6n"
server_key: "c0b5a9370613bd4d6c0096a25fe9009c"
server_key_iv: "87cbbb51aac43b654ad015f3c5b23845"
server_key_version: "encryptionkey@0"
I20260808 18:31:17.746069 13419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:17.757884 13419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:17.761142 13419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:17.761332 13419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:17.761927 13419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:31:17.763103 13464 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 18:31:17.767343 13419 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 18:31:17.767405 13419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260808 18:31:17.767448 13419 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 18:31:17.769070 13419 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 18:31:17.769127 13419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 18:31:17.769392 13464 tablet_bootstrap.cc:492] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2: Bootstrap starting.
I20260808 18:31:17.780107 13419 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:39627
I20260808 18:31:17.780243 13585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:39627 every 8 connection(s)
I20260808 18:31:17.781347 13419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:31:17.789536 13464 log.cc:824] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:17.791320 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 13419
I20260808 18:31:17.791816 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:38867
--local_ip_for_outbound_sockets=127.29.78.130
--tserver_master_addrs=127.29.78.190:45639
--webserver_port=40395
--webserver_interface=127.29.78.130
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 18:31:17 dist-test-slave-gp6n krb5kdc[12721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213877, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:31:17.797683 13464 tablet_bootstrap.cc:492] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:31:17.798264 13464 tablet_bootstrap.cc:492] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2: Bootstrap complete.
I20260808 18:31:17.798564 13588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:17.783486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:33417 (local address 127.29.78.190:45639)
0808 18:31:17.783838 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:17.783854 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:17.784266 (+ 412us) server_negotiation.cc:407] Connection header received
0808 18:31:17.785349 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:17.785359 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:17.785653 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:17.785974 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:17.787263 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.788216 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.789385 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.789914 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.790098 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:17.792806 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:17.792810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:17.792822 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:17.792850 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:17.795080 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:17.795719 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:17.795724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:17.795728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:17.795849 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:17.796253 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:17.796255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:17.796256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:17.796643 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:17.796793 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:17.797325 (+ 532us) server_negotiation.cc:299] Negotiation successful
0808 18:31:17.797598 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":128,"threads_started":1}
I20260808 18:31:17.798580 13589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:17.783512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.129:33417)
0808 18:31:17.784097 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:17.784138 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:17.784941 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:17.786041 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:17.786048 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:17.786595 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:17.787078 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:17.787092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.788381 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.788385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:17.789237 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:17.789246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.790064 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.790068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:17.790265 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:17.790954 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:17.790970 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:17.792606 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:17.795226 (+ 2620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:17.795232 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:17.795244 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:17.795583 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:17.795982 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:17.795985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:17.795987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:17.796155 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:17.796764 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:17.796770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:17.797026 (+ 256us) client_negotiation.cc:769] Sending connection context
0808 18:31:17.797214 (+ 188us) client_negotiation.cc:240] Negotiation successful
0808 18:31:17.797571 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":164,"threads_started":1}
I20260808 18:31:17.799784 13464 ts_tablet_manager.cc:1397] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2: Time spent bootstrapping tablet: real 0.031s user 0.020s sys 0.008s
I20260808 18:31:17.800906 13586 heartbeater.cc:347] Connected to a master server at 127.29.78.190:45639
I20260808 18:31:17.801141 13586 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:17.801626 13586 heartbeater.cc:511] Master 127.29.78.190:45639 requested a full tablet report, sending...
I20260808 18:31:17.803975 13369 ts_manager.cc:194] Registered new tserver with Master: 7d40e86dd6fe451aad3a8c90a4bef3c2 (127.29.78.129:39627)
I20260808 18:31:17.803720 13464 raft_consensus.cc:348] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } }
I20260808 18:31:17.804388 13464 raft_consensus.cc:729] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d40e86dd6fe451aad3a8c90a4bef3c2, State: Initialized, Role: FOLLOWER
I20260808 18:31:17.804936 13464 consensus_queue.cc:260] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } }
I20260808 18:31:17.806107 13464 ts_tablet_manager.cc:1428] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 18:31:17.807647 13369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:33417
I20260808 18:31:17.810137 13586 heartbeater.cc:503] Master 127.29.78.190:45639 was elected leader, sending a full tablet report...
Aug 08 18:31:17 dist-test-slave-gp6n krb5kdc[12721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213877, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260808 18:31:17.817952 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:17.807112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49768 (local address 127.29.78.129:39627)
0808 18:31:17.807471 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:17.807482 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:17.807876 (+ 394us) server_negotiation.cc:407] Connection header received
0808 18:31:17.808063 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:17.808068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:17.808256 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:17.808401 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:17.809104 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.810131 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.811402 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.811862 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.811945 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:17.814293 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:17.814307 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:17.814316 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:17.814337 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:17.816036 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:17.816542 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:17.816547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:17.816551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:17.816636 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:17.816925 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:17.816929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:17.816931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:17.817169 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:17.817312 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:17.817653 (+ 341us) server_negotiation.cc:299] Negotiation successful
0808 18:31:17.817813 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":108,"threads_started":1}
I20260808 18:31:17.817951 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:17.807015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:39627 (local address 127.0.0.1:49768)
0808 18:31:17.807725 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:17.807775 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:17.807915 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:17.808414 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:17.808419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:17.808846 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:17.808974 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:17.808982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.810251 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.810255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:17.811271 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:17.811282 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.811972 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.811976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:17.812090 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:17.812617 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:17.812631 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:17.814150 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:17.816164 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:17.816173 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:17.816178 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:17.816418 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:17.816734 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:17.816737 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:17.816739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:17.816832 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:17.817288 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:17.817295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:17.817421 (+ 126us) client_negotiation.cc:769] Sending connection context
0808 18:31:17.817560 (+ 139us) client_negotiation.cc:240] Negotiation successful
0808 18:31:17.817765 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":91,"threads_started":1}
I20260808 18:31:17.819216 13512 tablet_service.cc:1514] Processing DeleteTablet for tablet 43222644e66048e28f929dcecd716be8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:31:17 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:49768
I20260808 18:31:17.820086 13596 tablet_replica.cc:330] stopping tablet replica
I20260808 18:31:17.820413 13596 raft_consensus.cc:2229] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:31:17.820677 13596 raft_consensus.cc:2258] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:31:17.821772 13596 ts_tablet_manager.cc:1905] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:31:17.826363 13596 ts_tablet_manager.cc:1918] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:31:17.826498 13596 log.cc:1196] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/wal/wals/43222644e66048e28f929dcecd716be8
I20260808 18:31:17.826818 13596 ts_tablet_manager.cc:1939] T 43222644e66048e28f929dcecd716be8 P 7d40e86dd6fe451aad3a8c90a4bef3c2: Deleting consensus metadata
I20260808 18:31:17.828800 13353 catalog_manager.cc:4841] TS 7d40e86dd6fe451aad3a8c90a4bef3c2 (127.29.78.129:39627): tablet 43222644e66048e28f929dcecd716be8 (table test-table [id=7993d379a8044f73b3f092d2b4d0e33e]) successfully deleted
W20260808 18:31:17.914153 13591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:17.914387 13591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:17.914499 13591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:17.918306 13591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:17.918352 13591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:17.918457 13591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:31:17.923480 13591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:38867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=40395
--tserver_master_addrs=127.29.78.190:45639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:17.924412 13591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:17.925346 13591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:17.935056 13604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:17.935097 13601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:17.935228 13591 server_base.cc:1034] running on GCE node
W20260808 18:31:17.935290 13602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:17.935781 13591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:17.936434 13591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:17.937615 13591 hybrid_clock.cc:648] HybridClock initialized: now 1786213877937601 us; error 24 us; skew 500 ppm
Aug 08 18:31:17 dist-test-slave-gp6n krb5kdc[12721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213877, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:17.940336 13591 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:31:17.941222 13591 webserver.cc:460] Webserver started at http://127.29.78.130:40395/ using document root <none> and password file <none>
I20260808 18:31:17.941782 13591 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:17.941854 13591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:17.945503 13591 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:31:17.947445 13612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:17.948581 13591 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:31:17.948671 13591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "09ca2a2882524822841a90e33b0d8ad8"
format_stamp: "Formatted at 2026-08-08 18:31:16 on dist-test-slave-gp6n"
server_key: "6628d4e8c4e24dff6a637a7db5055565"
server_key_iv: "c1666259adfc9eb05bdd79e2c7809df9"
server_key_version: "encryptionkey@0"
I20260808 18:31:17.949259 13591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:17.968951 13591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:17.971869 13591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:17.972033 13591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:17.972560 13591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:31:17.973333 13591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:31:17.973380 13591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:17.973420 13591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:31:17.973443 13591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:17.982888 13591 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:38867
I20260808 18:31:17.982916 13742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:38867 every 8 connection(s)
I20260808 18:31:17.983891 13591 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:31:17.991176 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 13591
I20260808 18:31:17.991652 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:36891
--local_ip_for_outbound_sockets=127.29.78.131
--tserver_master_addrs=127.29.78.190:45639
--webserver_port=38181
--webserver_interface=127.29.78.131
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 18:31:17 dist-test-slave-gp6n krb5kdc[12721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213877, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:31:17.997555 13588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:17.985702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:54701 (local address 127.29.78.190:45639)
0808 18:31:17.985871 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:17.985876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:17.986745 (+ 869us) server_negotiation.cc:407] Connection header received
0808 18:31:17.988174 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:17.988177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:17.988227 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:17.988303 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:17.989440 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.989962 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.991154 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.991518 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.991632 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:17.994124 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:17.994127 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:17.994129 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:17.994150 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:17.995555 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:17.996051 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:17.996056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:17.996058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:17.996140 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:17.996471 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:17.996475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:17.996477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:17.996655 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:17.996764 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:17.997239 (+ 475us) server_negotiation.cc:299] Negotiation successful
0808 18:31:17.997378 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 18:31:17.998322 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:17.986092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.130:54701)
0808 18:31:17.986593 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:17.986636 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:17.987983 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:17.988468 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:17.988487 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:17.988838 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:17.989297 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:17.989327 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.990067 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.990069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:17.991032 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:17.991041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:17.991665 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:17.991667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:17.991878 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:17.992499 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:17.992515 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:17.993947 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:17.995675 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:17.995680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:17.995694 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:17.995946 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:17.996233 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:17.996237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:17.996239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:17.996375 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:17.996763 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:17.996770 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:17.997010 (+ 240us) client_negotiation.cc:769] Sending connection context
0808 18:31:17.997197 (+ 187us) client_negotiation.cc:240] Negotiation successful
0808 18:31:17.997516 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":117,"threads_started":1}
I20260808 18:31:17.999567 13743 heartbeater.cc:347] Connected to a master server at 127.29.78.190:45639
I20260808 18:31:17.999804 13743 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:18.000311 13743 heartbeater.cc:511] Master 127.29.78.190:45639 requested a full tablet report, sending...
I20260808 18:31:18.001374 13369 ts_manager.cc:194] Registered new tserver with Master: 09ca2a2882524822841a90e33b0d8ad8 (127.29.78.130:38867)
I20260808 18:31:18.002254 13369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:54701
W20260808 18:31:18.103199 13746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:18.103466 13746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:18.103508 13746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:18.108155 13746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:18.108213 13746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:18.108270 13746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:31:18.113440 13746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:36891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=38181
--tserver_master_addrs=127.29.78.190:45639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:18.114424 13746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:18.115393 13746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:18.125087 13754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:18.125103 13751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:18.125103 13752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:18.125416 13746 server_base.cc:1034] running on GCE node
I20260808 18:31:18.125945 13746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:18.126700 13746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:18.127861 13746 hybrid_clock.cc:648] HybridClock initialized: now 1786213878127848 us; error 27 us; skew 500 ppm
Aug 08 18:31:18 dist-test-slave-gp6n krb5kdc[12721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213878, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:18.130734 13746 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:31:18.131672 13746 webserver.cc:460] Webserver started at http://127.29.78.131:38181/ using document root <none> and password file <none>
I20260808 18:31:18.132236 13746 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:18.132345 13746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:18.136125 13746 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260808 18:31:18.138229 13761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:18.139518 13746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:31:18.139652 13746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "801408a989244177ae8b1bd0a14daa61"
format_stamp: "Formatted at 2026-08-08 18:31:17 on dist-test-slave-gp6n"
server_key: "3bfce5bb4a85eb52eedd5e693feba061"
server_key_iv: "2baaa849bca8e70f93daa7a63f3726e9"
server_key_version: "encryptionkey@0"
I20260808 18:31:18.140198 13746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:18.155125 13746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:18.158293 13746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:18.158521 13746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:18.159125 13746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:31:18.159924 13746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:31:18.160010 13746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:18.160102 13746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:31:18.160152 13746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:18.169503 13746 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:36891
I20260808 18:31:18.169534 13887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:36891 every 8 connection(s)
I20260808 18:31:18.170540 13746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:31:18.179144 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 13746
Aug 08 18:31:18 dist-test-slave-gp6n krb5kdc[12721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213878, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:31:18.184368 13588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:18.172249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:47363 (local address 127.29.78.190:45639)
0808 18:31:18.172394 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:18.172398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:18.173242 (+ 844us) server_negotiation.cc:407] Connection header received
0808 18:31:18.174624 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:18.174627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:18.174686 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:18.174794 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:18.175937 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.176464 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.177619 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.177847 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.177962 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:18.180645 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:18.180649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:18.180652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:18.180685 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:18.182173 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:18.182742 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:18.182747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:18.182750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:18.182834 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:18.183183 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:18.183186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:18.183188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:18.183425 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:18.183543 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:18.184071 (+ 528us) server_negotiation.cc:299] Negotiation successful
0808 18:31:18.184194 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":61}
I20260808 18:31:18.185359 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:18.172637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.131:47363)
0808 18:31:18.173082 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:18.173123 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:18.174389 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:18.174966 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:18.174978 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:18.175327 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:18.175782 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:18.175796 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.176615 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.176619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:18.177489 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:18.177498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.178007 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.178011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:18.178176 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:18.178813 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:18.178828 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:18.180474 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:18.182295 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:18.182300 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:18.182314 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:18.182638 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:18.182926 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:18.182931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:18.182933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:18.183078 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:18.183539 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:18.183545 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:18.183789 (+ 244us) client_negotiation.cc:769] Sending connection context
0808 18:31:18.184015 (+ 226us) client_negotiation.cc:240] Negotiation successful
0808 18:31:18.184320 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":110,"threads_started":1}
I20260808 18:31:18.186599 13888 heartbeater.cc:347] Connected to a master server at 127.29.78.190:45639
I20260808 18:31:18.186820 13888 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:18.187352 13888 heartbeater.cc:511] Master 127.29.78.190:45639 requested a full tablet report, sending...
I20260808 18:31:18.187958 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:18.179832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43940 (local address 127.29.78.190:45639)
0808 18:31:18.180076 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:18.180081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:18.180283 (+ 202us) server_negotiation.cc:407] Connection header received
0808 18:31:18.180454 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:18.180457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:18.180499 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:18.180592 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:18.181479 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.181991 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.182927 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.183162 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.183273 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:18.184662 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:18.184666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:18.184669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:18.184691 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:18.186160 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:18.186666 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:18.186671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:18.186674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:18.186766 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:18.187081 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:18.187085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:18.187087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:18.187315 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:18.187547 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:18.187680 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 18:31:18.187794 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":175,"spinlock_wait_cycles":34048,"thread_start_us":99,"threads_started":1}
I20260808 18:31:18.188400 13369 ts_manager.cc:194] Registered new tserver with Master: 801408a989244177ae8b1bd0a14daa61 (127.29.78.131:36891)
I20260808 18:31:18.189124 13369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:47363
I20260808 18:31:18.189760 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:31:18.193921 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:18.190441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43950 (local address 127.29.78.190:45639)
0808 18:31:18.190549 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:18.190553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:18.190688 (+ 135us) server_negotiation.cc:407] Connection header received
0808 18:31:18.190849 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:18.190851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:18.190927 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:18.191011 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:31:18.191267 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.191723 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.192653 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.192862 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.192940 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:18.193204 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:31:18.193542 (+ 338us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:31:18.193631 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:18.193740 (+ 109us) server_negotiation.cc:299] Negotiation successful
0808 18:31:18.193784 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 18:31:18.194609 13368 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:31:18.197657 13368 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:31:18.205392 13512 tablet_service.cc:1467] Processing CreateTablet for tablet ab1194c39c3c442f9d3ff121233c2efe (DEFAULT_TABLE table=test-table [id=863ee4fcf5074ed9a977ddd7c39ee5b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:31:18.206080 13512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab1194c39c3c442f9d3ff121233c2efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:31:18.208762 13464 tablet_bootstrap.cc:492] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2: Bootstrap starting.
I20260808 18:31:18.209230 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:18.204193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:36891 (local address 127.0.0.1:36914)
0808 18:31:18.204656 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:18.204674 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:18.204759 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:18.205272 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:18.205275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:18.205292 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:18.205397 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:18.205403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.206781 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.206785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:18.207960 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:18.207969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.208822 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.208825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:18.208928 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:18.208977 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 18:31:18.209034 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 18:31:18.209108 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393}
I20260808 18:31:18.209687 13464 tablet_bootstrap.cc:654] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2: Neither blocks nor log segments found. Creating new log.
I20260808 18:31:18.209863 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:18.204330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36914 (local address 127.29.78.131:36891)
0808 18:31:18.204696 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:18.204707 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:18.204731 (+ 24us) server_negotiation.cc:407] Connection header received
0808 18:31:18.204885 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:18.204891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:18.205102 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:18.205262 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:18.205518 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.206640 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.208077 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.208677 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.209413 (+ 736us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:18.209559 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:18.209614 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 18:31:18.209721 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":101,"threads_started":1}
I20260808 18:31:18.210211 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:18.204437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:38867 (local address 127.0.0.1:51236)
0808 18:31:18.204978 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:18.204991 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:18.205067 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:18.205790 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:18.205793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:18.205811 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:18.205912 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:18.205917 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.207807 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.207811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:18.208883 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:18.208893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.209827 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.209830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:18.209916 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:18.209937 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 18:31:18.210019 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 18:31:18.210091 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":103,"threads_started":1}
I20260808 18:31:18.210261 13901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:18.204624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51236 (local address 127.29.78.130:38867)
0808 18:31:18.205311 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:18.205323 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:18.205344 (+ 21us) server_negotiation.cc:407] Connection header received
0808 18:31:18.205417 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:18.205424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:18.205630 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:18.206170 (+ 540us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:18.206333 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.207648 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.209024 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.209704 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.209825 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:18.209953 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:18.210044 (+ 91us) server_negotiation.cc:299] Negotiation successful
0808 18:31:18.210129 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":171,"threads_started":1}
I20260808 18:31:18.211544 13464 tablet_bootstrap.cc:492] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2: No bootstrap required, opened a new log
I20260808 18:31:18.211633 13464 ts_tablet_manager.cc:1397] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:31:18.212057 13464 raft_consensus.cc:348] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } }
I20260808 18:31:18.212188 13464 raft_consensus.cc:374] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:31:18.212225 13464 raft_consensus.cc:729] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d40e86dd6fe451aad3a8c90a4bef3c2, State: Initialized, Role: FOLLOWER
I20260808 18:31:18.212512 13464 consensus_queue.cc:260] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } }
I20260808 18:31:18.213377 13464 ts_tablet_manager.cc:1428] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 18:31:18.212984 13818 tablet_service.cc:1467] Processing CreateTablet for tablet ab1194c39c3c442f9d3ff121233c2efe (DEFAULT_TABLE table=test-table [id=863ee4fcf5074ed9a977ddd7c39ee5b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:31:18.213442 13663 tablet_service.cc:1467] Processing CreateTablet for tablet ab1194c39c3c442f9d3ff121233c2efe (DEFAULT_TABLE table=test-table [id=863ee4fcf5074ed9a977ddd7c39ee5b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:31:18.213970 13818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab1194c39c3c442f9d3ff121233c2efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:31:18.214502 13663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab1194c39c3c442f9d3ff121233c2efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:31:18.222615 13903 tablet_bootstrap.cc:492] T ab1194c39c3c442f9d3ff121233c2efe P 09ca2a2882524822841a90e33b0d8ad8: Bootstrap starting.
I20260808 18:31:18.223515 13904 tablet_bootstrap.cc:492] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61: Bootstrap starting.
I20260808 18:31:18.226289 13904 tablet_bootstrap.cc:654] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61: Neither blocks nor log segments found. Creating new log.
I20260808 18:31:18.226439 13903 tablet_bootstrap.cc:654] T ab1194c39c3c442f9d3ff121233c2efe P 09ca2a2882524822841a90e33b0d8ad8: Neither blocks nor log segments found. Creating new log.
I20260808 18:31:18.227087 13904 log.cc:824] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:18.227108 13903 log.cc:824] T ab1194c39c3c442f9d3ff121233c2efe P 09ca2a2882524822841a90e33b0d8ad8: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:18.228730 13903 tablet_bootstrap.cc:492] T ab1194c39c3c442f9d3ff121233c2efe P 09ca2a2882524822841a90e33b0d8ad8: No bootstrap required, opened a new log
I20260808 18:31:18.228742 13904 tablet_bootstrap.cc:492] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61: No bootstrap required, opened a new log
I20260808 18:31:18.228926 13903 ts_tablet_manager.cc:1397] T ab1194c39c3c442f9d3ff121233c2efe P 09ca2a2882524822841a90e33b0d8ad8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:31:18.228930 13904 ts_tablet_manager.cc:1397] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:31:18.232136 13903 raft_consensus.cc:348] T ab1194c39c3c442f9d3ff121233c2efe P 09ca2a2882524822841a90e33b0d8ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } }
I20260808 18:31:18.232378 13903 raft_consensus.cc:374] T ab1194c39c3c442f9d3ff121233c2efe P 09ca2a2882524822841a90e33b0d8ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:31:18.232460 13903 raft_consensus.cc:729] T ab1194c39c3c442f9d3ff121233c2efe P 09ca2a2882524822841a90e33b0d8ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ca2a2882524822841a90e33b0d8ad8, State: Initialized, Role: FOLLOWER
I20260808 18:31:18.232372 13904 raft_consensus.cc:348] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } }
I20260808 18:31:18.232698 13904 raft_consensus.cc:374] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:31:18.232759 13904 raft_consensus.cc:729] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 801408a989244177ae8b1bd0a14daa61, State: Initialized, Role: FOLLOWER
I20260808 18:31:18.232914 13903 consensus_queue.cc:260] T ab1194c39c3c442f9d3ff121233c2efe P 09ca2a2882524822841a90e33b0d8ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } }
I20260808 18:31:18.233232 13904 consensus_queue.cc:260] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } }
I20260808 18:31:18.233600 13743 heartbeater.cc:503] Master 127.29.78.190:45639 was elected leader, sending a full tablet report...
I20260808 18:31:18.233830 13888 heartbeater.cc:503] Master 127.29.78.190:45639 was elected leader, sending a full tablet report...
I20260808 18:31:18.234184 13904 ts_tablet_manager.cc:1428] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 18:31:18.234877 13903 ts_tablet_manager.cc:1428] T ab1194c39c3c442f9d3ff121233c2efe P 09ca2a2882524822841a90e33b0d8ad8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 18:31:18.260632 13908 raft_consensus.cc:482] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:31:18.260856 13908 raft_consensus.cc:504] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } }
I20260808 18:31:18.262205 13908 leader_election.cc:288] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ca2a2882524822841a90e33b0d8ad8 (127.29.78.130:38867), 7d40e86dd6fe451aad3a8c90a4bef3c2 (127.29.78.129:39627)
I20260808 18:31:18.267025 13901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:18.262506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:46987 (local address 127.29.78.130:38867)
0808 18:31:18.262645 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:18.262649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:18.262665 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:31:18.262823 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:18.262827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:18.262880 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:18.262997 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:18.263293 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.264101 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.265784 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.266386 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.266553 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:18.266607 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:18.266822 (+ 215us) server_negotiation.cc:299] Negotiation successful
0808 18:31:18.266892 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 18:31:18.267025 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:18.262641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:38725 (local address 127.29.78.129:39627)
0808 18:31:18.262775 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:18.262780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:18.262946 (+ 166us) server_negotiation.cc:407] Connection header received
0808 18:31:18.263116 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:18.263118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:18.263176 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:18.263288 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:18.263555 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.264283 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.265784 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.266460 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.266590 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:18.266725 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:18.266821 (+ 96us) server_negotiation.cc:299] Negotiation successful
0808 18:31:18.266892 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 18:31:18.267052 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:18.262399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:38867 (local address 127.29.78.131:46987)
0808 18:31:18.262577 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:18.262593 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:18.262681 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:18.263021 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:18.263027 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:18.263048 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:18.263169 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:18.263174 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.264216 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.264219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:18.265656 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:18.265665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.266521 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.266524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:18.266659 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:18.266721 (+ 62us) client_negotiation.cc:769] Sending connection context
0808 18:31:18.266794 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 18:31:18.266891 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":108}
I20260808 18:31:18.267064 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:18.262448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:39627 (local address 127.29.78.131:38725)
0808 18:31:18.262848 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:18.262859 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:18.262997 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:18.263317 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:18.263320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:18.263338 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:18.263424 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:18.263429 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.264379 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.264383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:18.265655 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:18.265665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.266586 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.266589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:18.266680 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:18.266717 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 18:31:18.266794 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 18:31:18.266891 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":94,"thread_start_us":141,"threads_started":1}
I20260808 18:31:18.267732 13691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1194c39c3c442f9d3ff121233c2efe" candidate_uuid: "801408a989244177ae8b1bd0a14daa61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ca2a2882524822841a90e33b0d8ad8" is_pre_election: true
I20260808 18:31:18.267719 13535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1194c39c3c442f9d3ff121233c2efe" candidate_uuid: "801408a989244177ae8b1bd0a14daa61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" is_pre_election: true
I20260808 18:31:18.268019 13535 raft_consensus.cc:2454] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 801408a989244177ae8b1bd0a14daa61 in term 0.
I20260808 18:31:18.268018 13691 raft_consensus.cc:2454] T ab1194c39c3c442f9d3ff121233c2efe P 09ca2a2882524822841a90e33b0d8ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 801408a989244177ae8b1bd0a14daa61 in term 0.
I20260808 18:31:18.268504 13764 leader_election.cc:302] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ca2a2882524822841a90e33b0d8ad8, 801408a989244177ae8b1bd0a14daa61; no voters:
I20260808 18:31:18.268754 13908 raft_consensus.cc:2781] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:31:18.268836 13908 raft_consensus.cc:482] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:31:18.268877 13908 raft_consensus.cc:3037] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:31:18.269936 13908 raft_consensus.cc:504] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } }
I20260808 18:31:18.270284 13908 leader_election.cc:288] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61 [CANDIDATE]: Term 1 election: Requested vote from peers 09ca2a2882524822841a90e33b0d8ad8 (127.29.78.130:38867), 7d40e86dd6fe451aad3a8c90a4bef3c2 (127.29.78.129:39627)
I20260808 18:31:18.270656 13691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1194c39c3c442f9d3ff121233c2efe" candidate_uuid: "801408a989244177ae8b1bd0a14daa61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ca2a2882524822841a90e33b0d8ad8"
I20260808 18:31:18.270704 13535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1194c39c3c442f9d3ff121233c2efe" candidate_uuid: "801408a989244177ae8b1bd0a14daa61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2"
I20260808 18:31:18.270761 13691 raft_consensus.cc:3037] T ab1194c39c3c442f9d3ff121233c2efe P 09ca2a2882524822841a90e33b0d8ad8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:31:18.270864 13535 raft_consensus.cc:3037] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:31:18.271826 13691 raft_consensus.cc:2454] T ab1194c39c3c442f9d3ff121233c2efe P 09ca2a2882524822841a90e33b0d8ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 801408a989244177ae8b1bd0a14daa61 in term 1.
I20260808 18:31:18.271927 13535 raft_consensus.cc:2454] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 801408a989244177ae8b1bd0a14daa61 in term 1.
I20260808 18:31:18.272224 13762 leader_election.cc:302] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d40e86dd6fe451aad3a8c90a4bef3c2, 801408a989244177ae8b1bd0a14daa61; no voters:
I20260808 18:31:18.272413 13908 raft_consensus.cc:2781] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:31:18.272681 13908 raft_consensus.cc:686] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61 [term 1 LEADER]: Becoming Leader. State: Replica: 801408a989244177ae8b1bd0a14daa61, State: Running, Role: LEADER
I20260808 18:31:18.273005 13908 consensus_queue.cc:237] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } }
I20260808 18:31:18.276198 13368 catalog_manager.cc:5495] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 801408a989244177ae8b1bd0a14daa61 (127.29.78.131). New cstate: current_term: 1 leader_uuid: "801408a989244177ae8b1bd0a14daa61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } health_report { overall_health: UNKNOWN } } }
W20260808 18:31:18.282572 13587 tablet.cc:2369] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:31:18.297191 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:18.293783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36916 (local address 127.29.78.131:36891)
0808 18:31:18.293907 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:18.293910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:18.293923 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:31:18.294065 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:18.294067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:18.294116 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:18.294199 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:31:18.294398 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.294951 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.295881 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:18.296094 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:18.296178 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:18.296453 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:31:18.296857 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:31:18.296951 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:18.297023 (+ 72us) server_negotiation.cc:299] Negotiation successful
0808 18:31:18.297071 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 18:31:18.302990 13535 raft_consensus.cc:1261] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Refusing update from remote peer 801408a989244177ae8b1bd0a14daa61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:31:18.303028 13691 raft_consensus.cc:1261] T ab1194c39c3c442f9d3ff121233c2efe P 09ca2a2882524822841a90e33b0d8ad8 [term 1 FOLLOWER]: Refusing update from remote peer 801408a989244177ae8b1bd0a14daa61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:31:18.303761 13908 consensus_queue.cc:1035] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:31:18.303957 13910 consensus_queue.cc:1035] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:31:18.308845 13912 mvcc.cc:205] Tried to move back new op lower bound from 7316332045521846272 to 7316332045409243136. Current Snapshot: MvccSnapshot[applied={T|T < 7316332045521846272}]
I20260808 18:31:18.311990 13913 mvcc.cc:205] Tried to move back new op lower bound from 7316332045521846272 to 7316332045409243136. Current Snapshot: MvccSnapshot[applied={T|T < 7316332045521846272}]
I20260808 18:31:18.319308 13368 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:31:18.319451 13368 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:31:18.321861 13368 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669: Sending DeleteTablet for 3 replicas of tablet ab1194c39c3c442f9d3ff121233c2efe
I20260808 18:31:18.322575 13818 tablet_service.cc:1514] Processing DeleteTablet for tablet ab1194c39c3c442f9d3ff121233c2efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:31:18 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:36914
I20260808 18:31:18.322820 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 13419
I20260808 18:31:18.323101 13663 tablet_service.cc:1514] Processing DeleteTablet for tablet ab1194c39c3c442f9d3ff121233c2efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:31:18 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:51236
I20260808 18:31:18.324937 13922 tablet_replica.cc:330] T ab1194c39c3c442f9d3ff121233c2efe P 09ca2a2882524822841a90e33b0d8ad8: stopping tablet replica
I20260808 18:31:18.325155 13921 tablet_replica.cc:330] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61: stopping tablet replica
I20260808 18:31:18.325378 13922 raft_consensus.cc:2229] T ab1194c39c3c442f9d3ff121233c2efe P 09ca2a2882524822841a90e33b0d8ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:31:18.325383 13921 raft_consensus.cc:2229] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:31:18.325737 13921 raft_consensus.cc:2258] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:31:18.325763 13922 raft_consensus.cc:2258] T ab1194c39c3c442f9d3ff121233c2efe P 09ca2a2882524822841a90e33b0d8ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:31:18.327366 13921 ts_tablet_manager.cc:1905] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:31:18.327366 13922 ts_tablet_manager.cc:1905] T ab1194c39c3c442f9d3ff121233c2efe P 09ca2a2882524822841a90e33b0d8ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:31:18.330161 13922 ts_tablet_manager.cc:1918] T ab1194c39c3c442f9d3ff121233c2efe P 09ca2a2882524822841a90e33b0d8ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:31:18.330161 13921 ts_tablet_manager.cc:1918] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:31:18.330262 13921 log.cc:1196] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/ab1194c39c3c442f9d3ff121233c2efe
I20260808 18:31:18.330427 13922 log.cc:1196] T ab1194c39c3c442f9d3ff121233c2efe P 09ca2a2882524822841a90e33b0d8ad8: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/ab1194c39c3c442f9d3ff121233c2efe
I20260808 18:31:18.330699 13921 ts_tablet_manager.cc:1939] T ab1194c39c3c442f9d3ff121233c2efe P 801408a989244177ae8b1bd0a14daa61: Deleting consensus metadata
I20260808 18:31:18.330929 13922 ts_tablet_manager.cc:1939] T ab1194c39c3c442f9d3ff121233c2efe P 09ca2a2882524822841a90e33b0d8ad8: Deleting consensus metadata
I20260808 18:31:18.332002 13355 catalog_manager.cc:4841] TS 09ca2a2882524822841a90e33b0d8ad8 (127.29.78.130:38867): tablet ab1194c39c3c442f9d3ff121233c2efe (table test-table [id=863ee4fcf5074ed9a977ddd7c39ee5b7]) successfully deleted
I20260808 18:31:18.332003 13356 catalog_manager.cc:4841] TS 801408a989244177ae8b1bd0a14daa61 (127.29.78.131:36891): tablet ab1194c39c3c442f9d3ff121233c2efe (table test-table [id=863ee4fcf5074ed9a977ddd7c39ee5b7]) successfully deleted
W20260808 18:31:18.335059 13353 connection.cc:535] client connection to 127.29.78.129:39627 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 18:31:18.335172 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 13591
W20260808 18:31:18.335208 13353 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 18:31:18.339427 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:18.339055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:39627 (local address 127.0.0.1:49778)
0808 18:31:18.339226 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:18.339314 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260808 18:31:18.339689 13353 catalog_manager.cc:4584] TS 7d40e86dd6fe451aad3a8c90a4bef3c2 (127.29.78.129:39627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab1194c39c3c442f9d3ff121233c2efe: Network error: Client connection negotiation failed: client connection to 127.29.78.129:39627: connect: Connection refused (error 111)
I20260808 18:31:18.343247 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 13746
I20260808 18:31:18.350751 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 13333
I20260808 18:31:18.358739 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:45639
--webserver_interface=127.29.78.190
--webserver_port=41443
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:45639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:31:18.473302 13924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:18.473594 13924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:18.473664 13924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:18.477739 13924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:31:18.477833 13924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:18.477867 13924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:18.477890 13924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:31:18.477931 13924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:31:18.483129 13924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:45639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:45639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=41443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:18.484195 13924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:18.485275 13924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:18.493718 13933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:18.493820 13924 server_base.cc:1034] running on GCE node
W20260808 18:31:18.493768 13930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:18.493749 13931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:18.494765 13924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:18.495829 13924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:18.497033 13924 hybrid_clock.cc:648] HybridClock initialized: now 1786213878496993 us; error 45 us; skew 500 ppm
Aug 08 18:31:18 dist-test-slave-gp6n krb5kdc[12721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213878, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:18.500211 13924 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:31:18.501156 13924 webserver.cc:460] Webserver started at http://127.29.78.190:41443/ using document root <none> and password file <none>
I20260808 18:31:18.501693 13924 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:18.501760 13924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:18.505801 13924 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:31:18.508106 13940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:18.509487 13924 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:31:18.509639 13924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "38f4095c21994a79abd088e4f600b669"
format_stamp: "Formatted at 2026-08-08 18:31:16 on dist-test-slave-gp6n"
server_key: "04084db741c8336a1733612f86f37336"
server_key_iv: "9bd326816e3a4ebafae7bcd58401684e"
server_key_version: "encryptionkey@0"
I20260808 18:31:18.510231 13924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:18.540403 13924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:18.543697 13924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:18.543919 13924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:18.551959 13924 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:45639
I20260808 18:31:18.551995 14003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:45639 every 8 connection(s)
I20260808 18:31:18.553078 13924 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:31:18.556159 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 13924
I20260808 18:31:18.556705 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:39627
--local_ip_for_outbound_sockets=127.29.78.129
--tserver_master_addrs=127.29.78.190:45639
--webserver_port=42479
--webserver_interface=127.29.78.129
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:31:18.557927 14004 sys_catalog.cc:263] Verifying existing consensus state
I20260808 18:31:18.560017 14004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669: Bootstrap starting.
I20260808 18:31:18.578914 14004 log.cc:824] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:18.591051 14004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:31:18.591477 14004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669: Bootstrap complete.
I20260808 18:31:18.596268 14004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f4095c21994a79abd088e4f600b669" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45639 } }
I20260808 18:31:18.596746 14004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f4095c21994a79abd088e4f600b669, State: Initialized, Role: FOLLOWER
I20260808 18:31:18.597275 14004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f4095c21994a79abd088e4f600b669" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45639 } }
I20260808 18:31:18.597441 14004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:31:18.597537 14004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:31:18.597666 14004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 2 FOLLOWER]: Advancing to term 3
I20260808 18:31:18.599937 14004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f4095c21994a79abd088e4f600b669" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45639 } }
I20260808 18:31:18.600288 14004 leader_election.cc:302] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38f4095c21994a79abd088e4f600b669; no voters:
I20260808 18:31:18.600656 14004 leader_election.cc:288] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 18:31:18.600750 14008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 18:31:18.600978 14008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [term 3 LEADER]: Becoming Leader. State: Replica: 38f4095c21994a79abd088e4f600b669, State: Running, Role: LEADER
I20260808 18:31:18.601289 14008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f4095c21994a79abd088e4f600b669" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45639 } }
I20260808 18:31:18.602025 14004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:31:18.602962 14009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "38f4095c21994a79abd088e4f600b669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f4095c21994a79abd088e4f600b669" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45639 } } }
I20260808 18:31:18.603130 14009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [sys.catalog]: This master's current role is: LEADER
I20260808 18:31:18.603469 14010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38f4095c21994a79abd088e4f600b669. Latest consensus state: current_term: 3 leader_uuid: "38f4095c21994a79abd088e4f600b669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f4095c21994a79abd088e4f600b669" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45639 } } }
I20260808 18:31:18.603577 14010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669 [sys.catalog]: This master's current role is: LEADER
I20260808 18:31:18.607481 14025 catalog_manager.cc:1259] Loaded cluster ID: 699d79ad19fc4cc382279440dfc7bd53
I20260808 18:31:18.607563 14025 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669: loading cluster ID for follower catalog manager: success
W20260808 18:31:18.608512 14025 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 38f4095c21994a79abd088e4f600b669: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260808 18:31:18.608601 14025 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260808 18:31:18.608680 14019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:31:18.611521 14019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:31:18.612078 14019 catalog_manager.cc:1259] Loaded cluster ID: 699d79ad19fc4cc382279440dfc7bd53
I20260808 18:31:18.612239 14019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260808 18:31:18.612903 14019 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fee64fab9ad google::LogMessage::Fail() at ??:0
@ 0x7fee64fae88a google::LogMessage::SendToLog() at ??:0
@ 0x7fee64fab4de google::LogMessage::Flush() at ??:0
@ 0x7fee64faca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fee6f8eafdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fee6f8ebd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fee6f8e8579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fee6f92c640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fee671fd3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fee671fdc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fee671ff3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fee671ee512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fee688326db start_thread at ??:0
@ 0x7fee63aa961f clone at ??:0
W20260808 18:31:18.713456 14006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:18.713817 14006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:18.713893 14006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:18.720077 14006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:18.720155 14006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:18.720237 14006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:31:18.726125 14006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:39627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=42479
--tserver_master_addrs=127.29.78.190:45639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:18.727236 14006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:18.728269 14006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:18.738016 14056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:18.738049 14054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:18.738529 14006 server_base.cc:1034] running on GCE node
W20260808 18:31:18.738035 14059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:18.739014 14006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:18.739660 14006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:18.740862 14006 hybrid_clock.cc:648] HybridClock initialized: now 1786213878740838 us; error 42 us; skew 500 ppm
Aug 08 18:31:18 dist-test-slave-gp6n krb5kdc[12721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213878, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:18.743832 14006 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:31:18.744722 14006 webserver.cc:460] Webserver started at http://127.29.78.129:42479/ using document root <none> and password file <none>
I20260808 18:31:18.745275 14006 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:18.745344 14006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:18.749197 14006 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:31:18.751458 14066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:18.752811 14006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:31:18.752948 14006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2"
format_stamp: "Formatted at 2026-08-08 18:31:16 on dist-test-slave-gp6n"
server_key: "c0b5a9370613bd4d6c0096a25fe9009c"
server_key_iv: "87cbbb51aac43b654ad015f3c5b23845"
server_key_version: "encryptionkey@0"
I20260808 18:31:18.753507 14006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:18.770854 14006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:18.774112 14006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:18.774318 14006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:18.774966 14006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:31:18.776110 14075 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 18:31:18.781647 14006 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 18:31:18.781725 14006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260808 18:31:18.781802 14006 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 18:31:18.783500 14006 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 18:31:18.783571 14006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260808 18:31:18.783841 14075 tablet_bootstrap.cc:492] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2: Bootstrap starting.
I20260808 18:31:18.794319 14006 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:39627
I20260808 18:31:18.794617 14195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:39627 every 8 connection(s)
I20260808 18:31:18.795423 14006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:31:18.805227 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 14006
I20260808 18:31:18.805649 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:38867
--local_ip_for_outbound_sockets=127.29.78.130
--tserver_master_addrs=127.29.78.190:45639
--webserver_port=40395
--webserver_interface=127.29.78.130
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:31:18.806120 14075 log.cc:824] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:18.813238 14075 tablet_bootstrap.cc:492] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:31:18.813616 14075 tablet_bootstrap.cc:492] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2: Bootstrap complete.
I20260808 18:31:18.814788 14075 ts_tablet_manager.cc:1397] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.008s
I20260808 18:31:18.817058 14075 raft_consensus.cc:348] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } }
I20260808 18:31:18.817471 14075 raft_consensus.cc:729] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d40e86dd6fe451aad3a8c90a4bef3c2, State: Initialized, Role: FOLLOWER
I20260808 18:31:18.817950 14075 consensus_queue.cc:260] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } }
I20260808 18:31:18.818888 14075 ts_tablet_manager.cc:1428] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
W20260808 18:31:18.862259 14199 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:31:18.797416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.129:33653)
0808 18:31:18.797937 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:18.798001 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:18.798858 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:18.861414 (+ 62556us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":147,"threads_started":1}
W20260808 18:31:18.862769 14197 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:45639 (0 consecutive failures): Network error: Failed to ping master at 127.29.78.190:45639: Client connection negotiation failed: client connection to 127.29.78.190:45639: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 18:31:18.863554 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:18.863294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.129:58045)
0808 18:31:18.863421 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:18.863499 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 18:31:18.864265 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:18.864082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.129:34739)
0808 18:31:18.864177 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:18.864218 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 18:31:18.864516 14197 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 18:31:18.936992 14201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:18.937227 14201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:18.937309 14201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:18.941419 14201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:18.941480 14201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:18.941540 14201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:31:18.946777 14201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:38867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=40395
--tserver_master_addrs=127.29.78.190:45639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:18.947755 14201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:18.948752 14201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:18.958562 14207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:18.958698 14201 server_base.cc:1034] running on GCE node
W20260808 18:31:18.958636 14208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:18.958614 14210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:18.959194 14201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:18.959903 14201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:18.961089 14201 hybrid_clock.cc:648] HybridClock initialized: now 1786213878961074 us; error 24 us; skew 500 ppm
Aug 08 18:31:18 dist-test-slave-gp6n krb5kdc[12721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213878, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:18.963925 14201 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:31:18.964773 14201 webserver.cc:460] Webserver started at http://127.29.78.130:40395/ using document root <none> and password file <none>
I20260808 18:31:18.965328 14201 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:18.965421 14201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:18.969180 14201 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:31:18.971228 14217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:18.972353 14201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:31:18.972484 14201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "09ca2a2882524822841a90e33b0d8ad8"
format_stamp: "Formatted at 2026-08-08 18:31:16 on dist-test-slave-gp6n"
server_key: "6628d4e8c4e24dff6a637a7db5055565"
server_key_iv: "c1666259adfc9eb05bdd79e2c7809df9"
server_key_version: "encryptionkey@0"
I20260808 18:31:18.973032 14201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:18.992558 14201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:18.995543 14201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:18.995743 14201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:18.996351 14201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:31:18.997152 14201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:31:18.997222 14201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:18.997293 14201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:31:18.997346 14201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:19.006804 14201 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:38867
I20260808 18:31:19.006834 14341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:38867 every 8 connection(s)
I20260808 18:31:19.007786 14201 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:31:19.011536 14344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:19.009864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.130:41661)
0808 18:31:19.010333 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:19.010835 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":130,"threads_started":1}
W20260808 18:31:19.012065 14342 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:45639 (0 consecutive failures): Network error: Failed to ping master at 127.29.78.190:45639: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
I20260808 18:31:19.012696 14344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:19.012469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.130:35609)
0808 18:31:19.012579 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:19.012631 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 18:31:19.013336 14344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:19.013147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.130:53649)
0808 18:31:19.013245 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:19.013285 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 18:31:19.013520 14342 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 18:31:19.014009 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 14201
I20260808 18:31:19.014525 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:36891
--local_ip_for_outbound_sockets=127.29.78.131
--tserver_master_addrs=127.29.78.190:45639
--webserver_port=38181
--webserver_interface=127.29.78.131
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:31:19.129053 14345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:19.129333 14345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:19.129420 14345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:19.133754 14345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:19.133826 14345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:19.133932 14345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:31:19.139420 14345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:36891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=38181
--tserver_master_addrs=127.29.78.190:45639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:19.140478 14345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:19.141546 14345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:19.151378 14351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:19.151477 14353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:19.151372 14350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:19.151721 14345 server_base.cc:1034] running on GCE node
I20260808 18:31:19.152153 14345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:19.152855 14345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:19.154040 14345 hybrid_clock.cc:648] HybridClock initialized: now 1786213879154004 us; error 45 us; skew 500 ppm
Aug 08 18:31:19 dist-test-slave-gp6n krb5kdc[12721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213879, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:19.157104 14345 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:31:19.158067 14345 webserver.cc:460] Webserver started at http://127.29.78.131:38181/ using document root <none> and password file <none>
I20260808 18:31:19.158701 14345 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:19.158769 14345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:19.162694 14345 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:31:19.164853 14360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:19.166083 14345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:31:19.166225 14345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "801408a989244177ae8b1bd0a14daa61"
format_stamp: "Formatted at 2026-08-08 18:31:17 on dist-test-slave-gp6n"
server_key: "3bfce5bb4a85eb52eedd5e693feba061"
server_key_iv: "2baaa849bca8e70f93daa7a63f3726e9"
server_key_version: "encryptionkey@0"
I20260808 18:31:19.166860 14345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:19.189563 14345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:19.192654 14345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:19.192879 14345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:19.193508 14345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:31:19.194353 14345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:31:19.194448 14345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:19.194528 14345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:31:19.194582 14345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:19.203817 14345 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:36891
I20260808 18:31:19.203847 14483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:36891 every 8 connection(s)
I20260808 18:31:19.204834 14345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:31:19.208467 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:19.206846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.131:34499)
0808 18:31:19.207320 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:19.207762 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":113,"threads_started":1}
W20260808 18:31:19.208880 14484 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:45639 (0 consecutive failures): Network error: Failed to ping master at 127.29.78.190:45639: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
I20260808 18:31:19.209420 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:19.209216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.131:36663)
0808 18:31:19.209324 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:19.209369 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260808 18:31:19.209990 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:19.209787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.131:52211)
0808 18:31:19.209897 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:19.209936 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260808 18:31:19.210193 14484 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 18:31:19.211544 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 14345
I20260808 18:31:19.865562 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:19.865090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.129:37841)
0808 18:31:19.865353 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:19.865415 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":73,"threads_started":1}
I20260808 18:31:20.014552 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:20.014080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.130:47105)
0808 18:31:20.014328 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:20.014388 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":71,"threads_started":1}
I20260808 18:31:20.211149 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:20.210765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.131:54635)
0808 18:31:20.211002 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:20.211056 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":71,"threads_started":1}
I20260808 18:31:20.480934 14493 raft_consensus.cc:482] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:31:20.481122 14493 raft_consensus.cc:504] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } }
I20260808 18:31:20.482161 14493 leader_election.cc:288] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 801408a989244177ae8b1bd0a14daa61 (127.29.78.131:36891), 09ca2a2882524822841a90e33b0d8ad8 (127.29.78.130:38867)
Aug 08 18:31:20 dist-test-slave-gp6n krb5kdc[12721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213878, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.131@KRBTEST.COM
Aug 08 18:31:20 dist-test-slave-gp6n krb5kdc[12721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213878, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.130@KRBTEST.COM
I20260808 18:31:20.497452 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:20.482656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:40219 (local address 127.29.78.130:38867)
0808 18:31:20.483053 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:20.483074 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:20.483107 (+ 33us) server_negotiation.cc:407] Connection header received
0808 18:31:20.483800 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:20.483805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:20.484090 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:20.484386 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:20.485784 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:20.486820 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:20.488035 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:20.488480 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:20.488648 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:20.492026 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:20.492037 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:20.492055 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:20.492098 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:20.494070 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:20.494927 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:20.494934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:20.494940 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:20.495113 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:20.495581 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:20.495584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:20.495586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:20.496159 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:20.496302 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:20.496959 (+ 657us) server_negotiation.cc:299] Negotiation successful
0808 18:31:20.497259 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":61,"threads_started":1}
I20260808 18:31:20.497455 14497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:20.482587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:50227 (local address 127.29.78.131:36891)
0808 18:31:20.482872 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:20.482898 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:20.482947 (+ 49us) server_negotiation.cc:407] Connection header received
0808 18:31:20.483692 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:20.483698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:20.484037 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:20.484318 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:20.485787 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:20.486838 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:20.488000 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:20.488482 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:20.488649 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:20.491695 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:20.491701 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:20.491712 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:20.491740 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:20.494070 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:20.494927 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:20.494934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:20.494940 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:20.495113 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:20.495843 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:20.495847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:20.495849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:20.496557 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:20.496658 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:20.496981 (+ 323us) server_negotiation.cc:299] Negotiation successful
0808 18:31:20.497259 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":79,"threads_started":1}
I20260808 18:31:20.497423 14495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:20.482506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:38867 (local address 127.29.78.129:40219)
0808 18:31:20.483001 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:20.483015 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:20.483103 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:20.484400 (+ 1297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:20.484407 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:20.485033 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:20.485610 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:20.485627 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:20.486976 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:20.486980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:20.487882 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:20.487892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:20.488610 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:20.488614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:20.488836 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:20.489608 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:20.489620 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:20.491823 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:20.494230 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:20.494237 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:20.494253 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:20.494798 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:20.495220 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:20.495223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:20.495225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:20.495463 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:20.496299 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:20.496305 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:20.496568 (+ 263us) client_negotiation.cc:769] Sending connection context
0808 18:31:20.496777 (+ 209us) client_negotiation.cc:240] Negotiation successful
0808 18:31:20.497143 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":179,"spinlock_wait_cycles":43520,"thread_start_us":72,"threads_started":1}
I20260808 18:31:20.497423 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:20.482435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:36891 (local address 127.29.78.129:50227)
0808 18:31:20.482742 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:20.482757 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:20.482885 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:20.484405 (+ 1520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:20.484408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:20.485014 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:20.485610 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:20.485627 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:20.486989 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:20.486992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:20.487880 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:20.487890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:20.488606 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:20.488610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:20.488826 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:20.489560 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:20.489576 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:20.491441 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:20.494230 (+ 2789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:20.494241 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:20.494253 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:20.494798 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:20.495220 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:20.495223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:20.495225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:20.495490 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:20.496666 (+ 1176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:20.496669 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:20.496755 (+ 86us) client_negotiation.cc:769] Sending connection context
0808 18:31:20.496841 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 18:31:20.497196 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":230,"spinlock_wait_cycles":9984,"thread_start_us":78,"threads_started":1}
W20260808 18:31:20.500226 14072 leader_election.cc:341] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 801408a989244177ae8b1bd0a14daa61 (127.29.78.131:36891): Not found: Tablet not found: ab1194c39c3c442f9d3ff121233c2efe
W20260808 18:31:20.500371 14070 leader_election.cc:341] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09ca2a2882524822841a90e33b0d8ad8 (127.29.78.130:38867): Not found: Tablet not found: ab1194c39c3c442f9d3ff121233c2efe
I20260808 18:31:20.500495 14070 leader_election.cc:302] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d40e86dd6fe451aad3a8c90a4bef3c2; no voters: 09ca2a2882524822841a90e33b0d8ad8, 801408a989244177ae8b1bd0a14daa61
I20260808 18:31:20.500773 14493 raft_consensus.cc:2726] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 18:31:20.866703 14495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:20.866315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.129:58841)
0808 18:31:20.866558 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:20.866616 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260808 18:31:21.015700 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:21.015270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.130:45377)
0808 18:31:21.015557 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:21.015613 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":73,"threads_started":1}
I20260808 18:31:21.212162 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:21.211778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.131:51823)
0808 18:31:21.212023 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:21.212077 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
I20260808 18:31:21.867887 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:21.867445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.129:59795)
0808 18:31:21.867730 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:21.867786 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":73,"threads_started":1}
I20260808 18:31:22.016887 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:22.016448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.130:50555)
0808 18:31:22.016731 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:22.016784 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":103,"threads_started":1}
I20260808 18:31:22.213213 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:22.212842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.131:35719)
0808 18:31:22.213062 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:22.213112 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":70,"threads_started":1}
I20260808 18:31:22.542768 14505 raft_consensus.cc:482] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:31:22.542917 14505 raft_consensus.cc:504] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } }
I20260808 18:31:22.543337 14505 leader_election.cc:288] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 801408a989244177ae8b1bd0a14daa61 (127.29.78.131:36891), 09ca2a2882524822841a90e33b0d8ad8 (127.29.78.130:38867)
W20260808 18:31:22.544287 14072 leader_election.cc:341] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 801408a989244177ae8b1bd0a14daa61 (127.29.78.131:36891): Not found: Tablet not found: ab1194c39c3c442f9d3ff121233c2efe
W20260808 18:31:22.544430 14070 leader_election.cc:341] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09ca2a2882524822841a90e33b0d8ad8 (127.29.78.130:38867): Not found: Tablet not found: ab1194c39c3c442f9d3ff121233c2efe
I20260808 18:31:22.544534 14070 leader_election.cc:302] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d40e86dd6fe451aad3a8c90a4bef3c2; no voters: 09ca2a2882524822841a90e33b0d8ad8, 801408a989244177ae8b1bd0a14daa61
I20260808 18:31:22.544692 14505 raft_consensus.cc:2726] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 18:31:22.869122 14506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:22.868672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.129:38969)
0808 18:31:22.868959 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:22.869020 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":103,"threads_started":1}
I20260808 18:31:23.018090 14507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:23.017658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.130:49993)
0808 18:31:23.017943 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:23.017999 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":95,"threads_started":1}
I20260808 18:31:23.214349 14508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:23.213901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.131:39997)
0808 18:31:23.214200 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:23.214254 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":94,"threads_started":1}
I20260808 18:31:23.870790 14509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:23.870130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.129:40047)
0808 18:31:23.870574 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:23.870651 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":132,"threads_started":1}
I20260808 18:31:24.019296 14510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:24.018887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.130:51203)
0808 18:31:24.019146 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:24.019202 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":93,"threads_started":1}
I20260808 18:31:24.215631 14512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:24.215204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.131:48443)
0808 18:31:24.215459 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:24.215540 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":92,"threads_started":1}
I20260808 18:31:24.528043 14513 raft_consensus.cc:482] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:31:24.528218 14513 raft_consensus.cc:504] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } }
I20260808 18:31:24.528743 14513 leader_election.cc:288] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 801408a989244177ae8b1bd0a14daa61 (127.29.78.131:36891), 09ca2a2882524822841a90e33b0d8ad8 (127.29.78.130:38867)
W20260808 18:31:24.529629 14072 leader_election.cc:341] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 801408a989244177ae8b1bd0a14daa61 (127.29.78.131:36891): Not found: Tablet not found: ab1194c39c3c442f9d3ff121233c2efe
W20260808 18:31:24.529742 14070 leader_election.cc:341] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09ca2a2882524822841a90e33b0d8ad8 (127.29.78.130:38867): Not found: Tablet not found: ab1194c39c3c442f9d3ff121233c2efe
I20260808 18:31:24.529861 14070 leader_election.cc:302] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d40e86dd6fe451aad3a8c90a4bef3c2; no voters: 09ca2a2882524822841a90e33b0d8ad8, 801408a989244177ae8b1bd0a14daa61
I20260808 18:31:24.529992 14513 raft_consensus.cc:2726] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 18:31:24.871992 14514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:24.871520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.129:37909)
0808 18:31:24.871823 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:24.871884 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":89,"threads_started":1}
I20260808 18:31:25.020435 14515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:25.020023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.130:56463)
0808 18:31:25.020280 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:25.020343 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":94,"threads_started":1}
I20260808 18:31:25.216856 14517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:25.216438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.131:53003)
0808 18:31:25.216692 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:25.216753 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
I20260808 18:31:25.873169 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:25.872749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.129:40541)
0808 18:31:25.873013 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:25.873072 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260808 18:31:26.021844 14519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:26.021306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.130:49333)
0808 18:31:26.021664 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:26.021726 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":126,"threads_started":1}
I20260808 18:31:26.076519 14520 raft_consensus.cc:482] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:31:26.076653 14520 raft_consensus.cc:504] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } }
I20260808 18:31:26.077129 14520 leader_election.cc:288] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 801408a989244177ae8b1bd0a14daa61 (127.29.78.131:36891), 09ca2a2882524822841a90e33b0d8ad8 (127.29.78.130:38867)
W20260808 18:31:26.077953 14072 leader_election.cc:341] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 801408a989244177ae8b1bd0a14daa61 (127.29.78.131:36891): Not found: Tablet not found: ab1194c39c3c442f9d3ff121233c2efe
W20260808 18:31:26.078078 14070 leader_election.cc:341] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09ca2a2882524822841a90e33b0d8ad8 (127.29.78.130:38867): Not found: Tablet not found: ab1194c39c3c442f9d3ff121233c2efe
I20260808 18:31:26.078186 14070 leader_election.cc:302] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d40e86dd6fe451aad3a8c90a4bef3c2; no voters: 09ca2a2882524822841a90e33b0d8ad8, 801408a989244177ae8b1bd0a14daa61
I20260808 18:31:26.078342 14520 raft_consensus.cc:2726] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 18:31:26.218241 14521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:26.217662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.131:34925)
0808 18:31:26.218019 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:26.218109 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":94,"threads_started":1}
I20260808 18:31:26.874464 14522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:26.873957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.129:53827)
0808 18:31:26.874246 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:26.874311 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":80,"threads_started":1}
I20260808 18:31:27.023082 14523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:27.022663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.130:59523)
0808 18:31:27.022924 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:27.022985 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
I20260808 18:31:27.219431 14524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:27.219044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.131:36035)
0808 18:31:27.219278 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:27.219333 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":68,"threads_started":1}
I20260808 18:31:27.875936 14525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:27.875443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.129:52245)
0808 18:31:27.875752 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:27.875813 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":96,"threads_started":1}
I20260808 18:31:28.024238 14526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:28.023825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.130:56747)
0808 18:31:28.024083 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:28.024140 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
I20260808 18:31:28.220731 14527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:28.220219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.131:56945)
0808 18:31:28.220536 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:28.220611 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":74,"threads_started":1}
I20260808 18:31:28.804241 14528 raft_consensus.cc:482] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:31:28.804365 14528 raft_consensus.cc:504] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } }
I20260808 18:31:28.804857 14528 leader_election.cc:288] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 801408a989244177ae8b1bd0a14daa61 (127.29.78.131:36891), 09ca2a2882524822841a90e33b0d8ad8 (127.29.78.130:38867)
W20260808 18:31:28.805588 14072 leader_election.cc:341] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 801408a989244177ae8b1bd0a14daa61 (127.29.78.131:36891): Not found: Tablet not found: ab1194c39c3c442f9d3ff121233c2efe
W20260808 18:31:28.805676 14070 leader_election.cc:341] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09ca2a2882524822841a90e33b0d8ad8 (127.29.78.130:38867): Not found: Tablet not found: ab1194c39c3c442f9d3ff121233c2efe
I20260808 18:31:28.805769 14070 leader_election.cc:302] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d40e86dd6fe451aad3a8c90a4bef3c2; no voters: 09ca2a2882524822841a90e33b0d8ad8, 801408a989244177ae8b1bd0a14daa61
I20260808 18:31:28.805904 14528 raft_consensus.cc:2726] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 18:31:28.877180 14529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:28.876747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.129:57163)
0808 18:31:28.876996 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:28.877069 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260808 18:31:29.025628 14530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:29.025166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.130:49859)
0808 18:31:29.025478 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:29.025535 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":117,"threads_started":1}
I20260808 18:31:29.221975 14531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:29.221531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.131:57365)
0808 18:31:29.221828 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:29.221885 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":97,"threads_started":1}
I20260808 18:31:29.878378 14532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:29.877955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.129:43097)
0808 18:31:29.878220 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:29.878279 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
I20260808 18:31:30.026777 14533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:30.026314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.130:38241)
0808 18:31:30.026626 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:30.026683 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":137,"threads_started":1}
I20260808 18:31:30.223330 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:30.222776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.131:33087)
0808 18:31:30.223119 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:30.223195 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":120,"threads_started":1}
I20260808 18:31:30.879557 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:30.879133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.129:42159)
0808 18:31:30.879402 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:30.879459 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":96,"threads_started":1}
I20260808 18:31:31.028071 14536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:31.027648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.130:51301)
0808 18:31:31.027914 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:31.027973 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":74,"threads_started":1}
I20260808 18:31:31.224623 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:31.224173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.131:47903)
0808 18:31:31.224434 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:31.224519 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
I20260808 18:31:31.881011 14538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:31.880399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.129:54973)
0808 18:31:31.880722 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:31.880780 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":138,"threads_started":1}
I20260808 18:31:32.029544 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:32.029054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.130:43833)
0808 18:31:32.029380 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:32.029441 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":93,"threads_started":1}
I20260808 18:31:32.225785 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:32.225398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.131:52093)
0808 18:31:32.225634 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:32.225690 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":66,"threads_started":1}
I20260808 18:31:32.691365 14541 raft_consensus.cc:482] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:31:32.691498 14541 raft_consensus.cc:504] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801408a989244177ae8b1bd0a14daa61" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 36891 } } peers { permanent_uuid: "09ca2a2882524822841a90e33b0d8ad8" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 38867 } } peers { permanent_uuid: "7d40e86dd6fe451aad3a8c90a4bef3c2" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 39627 } }
I20260808 18:31:32.691968 14541 leader_election.cc:288] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 801408a989244177ae8b1bd0a14daa61 (127.29.78.131:36891), 09ca2a2882524822841a90e33b0d8ad8 (127.29.78.130:38867)
W20260808 18:31:32.692791 14072 leader_election.cc:341] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 801408a989244177ae8b1bd0a14daa61 (127.29.78.131:36891): Not found: Tablet not found: ab1194c39c3c442f9d3ff121233c2efe
W20260808 18:31:32.692937 14070 leader_election.cc:341] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09ca2a2882524822841a90e33b0d8ad8 (127.29.78.130:38867): Not found: Tablet not found: ab1194c39c3c442f9d3ff121233c2efe
I20260808 18:31:32.693037 14070 leader_election.cc:302] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d40e86dd6fe451aad3a8c90a4bef3c2; no voters: 09ca2a2882524822841a90e33b0d8ad8, 801408a989244177ae8b1bd0a14daa61
I20260808 18:31:32.693214 14541 raft_consensus.cc:2726] T ab1194c39c3c442f9d3ff121233c2efe P 7d40e86dd6fe451aad3a8c90a4bef3c2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 18:31:32.882211 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:32.881764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.129:33881)
0808 18:31:32.882052 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:32.882114 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":78,"threads_started":1}
I20260808 18:31:33.030853 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:33.030437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.130:53741)
0808 18:31:33.030703 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:33.030757 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":78,"threads_started":1}
I20260808 18:31:33.227070 14544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:33.226657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.131:37471)
0808 18:31:33.226907 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:33.226973 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
I20260808 18:31:33.883548 14546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:33.883162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.129:43721)
0808 18:31:33.883401 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:33.883455 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260808 18:31:34.032267 14547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:34.031777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.130:58911)
0808 18:31:34.032077 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:34.032167 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":89,"threads_started":1}
I20260808 18:31:34.212424 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 14006
I20260808 18:31:34.223196 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 14201
I20260808 18:31:34.228679 14548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:34.228148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45639 (local address 127.29.78.131:41353)
0808 18:31:34.228461 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:34.228536 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":82,"threads_started":1}
I20260808 18:31:34.234109 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 14345
2026-08-08T18:31:34Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19197 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:31:34 dist-test-slave-gp6n krb5kdc[14551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:31:34 dist-test-slave-gp6n krb5kdc[14551](info): set up 2 sockets
Aug 08 18:31:34 dist-test-slave-gp6n krb5kdc[14551](info): commencing operation
krb5kdc: starting...
W20260808 18:31:35.342567 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:31:35 dist-test-slave-gp6n krb5kdc[14551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:31:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:31:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:31:35.467355 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:44257
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:44257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:31:35.577986 14565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:35.578233 14565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:35.578270 14565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:35.582122 14565 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 18:31:35.582168 14565 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 18:31:35.582185 14565 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 18:31:35.582198 14565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:31:35.582216 14565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:35.582286 14565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:35.582315 14565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:31:35.582345 14565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:31:35.587409 14565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:44257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:44257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:35.588420 14565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:35.589468 14565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:31:35.598313 14565 server_base.cc:1034] running on GCE node
W20260808 18:31:35.598490 14570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:35.598330 14571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:35.598357 14573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:35.599308 14565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:35.600333 14565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:35.601497 14565 hybrid_clock.cc:648] HybridClock initialized: now 1786213895601480 us; error 31 us; skew 500 ppm
Aug 08 18:31:35 dist-test-slave-gp6n krb5kdc[14551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213895, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:35.604310 14565 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:31:35.605223 14565 webserver.cc:460] Webserver started at http://127.29.78.190:36637/ using document root <none> and password file <none>
I20260808 18:31:35.605790 14565 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:35.605856 14565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:35.606106 14565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:31:35.608168 14565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "047242e7bd634821bc96757a00bd1dea"
format_stamp: "Formatted at 2026-08-08 18:31:35 on dist-test-slave-gp6n"
server_key: "e3113693b33e4a523163d8e588301262"
server_key_iv: "33ff67cdf99bc2b4f26fc36f30049bc1"
server_key_version: "encryptionkey@0"
I20260808 18:31:35.608729 14565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "047242e7bd634821bc96757a00bd1dea"
format_stamp: "Formatted at 2026-08-08 18:31:35 on dist-test-slave-gp6n"
server_key: "e3113693b33e4a523163d8e588301262"
server_key_iv: "33ff67cdf99bc2b4f26fc36f30049bc1"
server_key_version: "encryptionkey@0"
I20260808 18:31:35.612389 14565 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 18:31:35.614713 14580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:35.615969 14565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:31:35.616147 14565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "047242e7bd634821bc96757a00bd1dea"
format_stamp: "Formatted at 2026-08-08 18:31:35 on dist-test-slave-gp6n"
server_key: "e3113693b33e4a523163d8e588301262"
server_key_iv: "33ff67cdf99bc2b4f26fc36f30049bc1"
server_key_version: "encryptionkey@0"
I20260808 18:31:35.616266 14565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:35.630838 14565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:35.633708 14565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:35.633930 14565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:35.641573 14565 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:44257
I20260808 18:31:35.641884 14637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:44257 every 8 connection(s)
I20260808 18:31:35.642611 14565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:31:35.643240 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 14565
I20260808 18:31:35.643383 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:31:35.643728 30010 external_mini_cluster.cc:1428] Setting key c93b1cb9991460781b49f2cfa21a3848
I20260808 18:31:35.645618 14638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:31:35.650978 14638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea: Bootstrap starting.
Aug 08 18:31:35 dist-test-slave-gp6n krb5kdc[14551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:31:35.653126 14638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea: Neither blocks nor log segments found. Creating new log.
I20260808 18:31:35.653978 14638 log.cc:824] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:35.655865 14638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea: No bootstrap required, opened a new log
I20260808 18:31:35.658190 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:35.645095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44918 (local address 127.29.78.190:44257)
0808 18:31:35.645551 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:35.645571 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:35.645816 (+ 245us) server_negotiation.cc:407] Connection header received
0808 18:31:35.646668 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:35.646685 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:35.647002 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:35.647341 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:35.648168 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:35.649150 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:35.650019 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:35.650546 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:35.650716 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:35.653178 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:35.653184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:35.653196 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:35.653244 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:35.655139 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:35.655575 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:35.655580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:35.655584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:35.655689 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:35.656021 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:35.656025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:35.656026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:35.656511 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:35.656692 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:35.656941 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 18:31:35.657227 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":147,"threads_started":1}
I20260808 18:31:35.659798 14638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047242e7bd634821bc96757a00bd1dea" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 44257 } }
I20260808 18:31:35.659997 14638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:31:35.660045 14638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 047242e7bd634821bc96757a00bd1dea, State: Initialized, Role: FOLLOWER
I20260808 18:31:35.660494 14638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047242e7bd634821bc96757a00bd1dea" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 44257 } }
I20260808 18:31:35.660614 14638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:31:35.660697 14638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:31:35.660820 14638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:31:35.661654 14638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047242e7bd634821bc96757a00bd1dea" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 44257 } }
I20260808 18:31:35.661975 14638 leader_election.cc:302] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 047242e7bd634821bc96757a00bd1dea; no voters:
I20260808 18:31:35.662253 14638 leader_election.cc:288] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:31:35.662384 14643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:31:35.662668 14643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 1 LEADER]: Becoming Leader. State: Replica: 047242e7bd634821bc96757a00bd1dea, State: Running, Role: LEADER
I20260808 18:31:35.662971 14643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047242e7bd634821bc96757a00bd1dea" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 44257 } }
I20260808 18:31:35.663450 14638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:31:35.664512 14645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "047242e7bd634821bc96757a00bd1dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047242e7bd634821bc96757a00bd1dea" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 44257 } } }
I20260808 18:31:35.664646 14645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [sys.catalog]: This master's current role is: LEADER
I20260808 18:31:35.664770 14644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 047242e7bd634821bc96757a00bd1dea. Latest consensus state: current_term: 1 leader_uuid: "047242e7bd634821bc96757a00bd1dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047242e7bd634821bc96757a00bd1dea" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 44257 } } }
I20260808 18:31:35.665376 14644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [sys.catalog]: This master's current role is: LEADER
W20260808 18:31:35.668241 14659 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:31:35.668330 14659 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:31:35.668411 14653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:31:35.669145 14653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:31:35.675004 14653 catalog_manager.cc:1347] Generated new cluster ID: b2d9f365084e4c51b109660263e5577a
I20260808 18:31:35.675064 14653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:31:35.704761 14653 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:31:35.706043 14653 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:31:35.713080 14653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea: Generated new TSK 0
I20260808 18:31:35.713840 14653 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260808 18:31:35.743774 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:44257
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:31:35.856683 14668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:35.856962 14668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:35.857007 14668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:35.861320 14668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:35.861387 14668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:35.861505 14668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:31:35.866958 14668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:44257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:35.868010 14668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:35.869026 14668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:35.878791 14674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:35.878983 14676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:35.878880 14668 server_base.cc:1034] running on GCE node
W20260808 18:31:35.878791 14673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:35.879483 14668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:35.880182 14668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:35.881410 14668 hybrid_clock.cc:648] HybridClock initialized: now 1786213895881390 us; error 31 us; skew 500 ppm
Aug 08 18:31:35 dist-test-slave-gp6n krb5kdc[14551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213895, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:35.884222 14668 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:31:35.885087 14668 webserver.cc:460] Webserver started at http://127.29.78.129:37093/ using document root <none> and password file <none>
I20260808 18:31:35.885658 14668 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:35.885756 14668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:35.885989 14668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:31:35.888049 14668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "78d85fdfa3db4e9abda1a406fbe28b73"
format_stamp: "Formatted at 2026-08-08 18:31:35 on dist-test-slave-gp6n"
server_key: "92e8ff621682108b6b24496ea7cbc2a1"
server_key_iv: "193b142c1770fc53af6b725ce3b52d44"
server_key_version: "encryptionkey@0"
I20260808 18:31:35.888624 14668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "78d85fdfa3db4e9abda1a406fbe28b73"
format_stamp: "Formatted at 2026-08-08 18:31:35 on dist-test-slave-gp6n"
server_key: "92e8ff621682108b6b24496ea7cbc2a1"
server_key_iv: "193b142c1770fc53af6b725ce3b52d44"
server_key_version: "encryptionkey@0"
I20260808 18:31:35.892547 14668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260808 18:31:35.894896 14686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:35.896044 14668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:31:35.896185 14668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "78d85fdfa3db4e9abda1a406fbe28b73"
format_stamp: "Formatted at 2026-08-08 18:31:35 on dist-test-slave-gp6n"
server_key: "92e8ff621682108b6b24496ea7cbc2a1"
server_key_iv: "193b142c1770fc53af6b725ce3b52d44"
server_key_version: "encryptionkey@0"
I20260808 18:31:35.896297 14668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:35.921640 14668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:35.924748 14668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:35.924958 14668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:35.925572 14668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:31:35.926719 14668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:31:35.926780 14668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:35.926843 14668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:31:35.926874 14668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:35.936472 14668 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:44659
I20260808 18:31:35.936497 14806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:44659 every 8 connection(s)
I20260808 18:31:35.937407 14668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:31:35.940645 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 14668
I20260808 18:31:35.940773 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:31:35.941102 30010 external_mini_cluster.cc:1428] Setting key b8c2d5483ca83aa1410e63448de1e88b
Aug 08 18:31:35 dist-test-slave-gp6n krb5kdc[14551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213895, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260808 18:31:35.952637 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:35.939258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:38289 (local address 127.29.78.190:44257)
0808 18:31:35.939394 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:35.939397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:35.940633 (+ 1236us) server_negotiation.cc:407] Connection header received
0808 18:31:35.941723 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:35.941727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:35.941781 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:35.941882 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:35.943517 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:35.944056 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:35.945494 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:35.945736 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:35.945853 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:35.948688 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:35.948695 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:35.948700 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:35.948736 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:35.950503 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:35.951000 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:35.951004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:35.951006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:35.951106 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:35.951442 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:35.951449 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:35.951454 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:35.951731 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:35.951843 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:35.952373 (+ 530us) server_negotiation.cc:299] Negotiation successful
0808 18:31:35.952509 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 18:31:35.953409 14809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:35.939600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.129:38289)
0808 18:31:35.940470 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:35.940514 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:35.941472 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:35.942132 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:35.942139 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:35.942673 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:35.943348 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:35.943371 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:35.944194 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:35.944200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:35.945351 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:35.945361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:35.945893 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:35.945897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:35.946077 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:35.946732 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:35.946748 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:35.948507 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:35.950631 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:35.950635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:35.950645 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:35.950893 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:35.951211 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:35.951213 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:35.951214 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:35.951340 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:35.951842 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:35.951846 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:35.952075 (+ 229us) client_negotiation.cc:769] Sending connection context
0808 18:31:35.952283 (+ 208us) client_negotiation.cc:240] Negotiation successful
0808 18:31:35.952531 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":543,"threads_started":1}
I20260808 18:31:35.954783 14807 heartbeater.cc:347] Connected to a master server at 127.29.78.190:44257
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
I20260808 18:31:35.955065 14807 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:35.955660 14807 heartbeater.cc:511] Master 127.29.78.190:44257 requested a full tablet report, sending...
I20260808 18:31:35.957262 14598 ts_manager.cc:194] Registered new tserver with Master: 78d85fdfa3db4e9abda1a406fbe28b73 (127.29.78.129:44659)
I20260808 18:31:35.958376 14598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:38289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260808 18:31:35.971573 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:44257
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:31:36.085392 14812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:36.085650 14812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:36.085693 14812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:36.089622 14812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:36.089684 14812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:36.089742 14812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:31:36.094905 14812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:44257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:36.095911 14812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:36.096891 14812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:36.106827 14817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:36.106912 14812 server_base.cc:1034] running on GCE node
W20260808 18:31:36.106829 14820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:36.106829 14818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:36.107524 14812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:36.108182 14812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:36.109368 14812 hybrid_clock.cc:648] HybridClock initialized: now 1786213896109345 us; error 42 us; skew 500 ppm
Aug 08 18:31:36 dist-test-slave-gp6n krb5kdc[14551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213896, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:36.112119 14812 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:31:36.112987 14812 webserver.cc:460] Webserver started at http://127.29.78.130:34837/ using document root <none> and password file <none>
I20260808 18:31:36.113561 14812 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:36.113653 14812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:36.113884 14812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:31:36.115983 14812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "08c941ac7e764e27abd46653e2b8c323"
format_stamp: "Formatted at 2026-08-08 18:31:36 on dist-test-slave-gp6n"
server_key: "ee1d5edf3ef279a2eb30fcd7aab35409"
server_key_iv: "3a0ff046b1decd1f1859f7d66c5673b6"
server_key_version: "encryptionkey@0"
I20260808 18:31:36.116572 14812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "08c941ac7e764e27abd46653e2b8c323"
format_stamp: "Formatted at 2026-08-08 18:31:36 on dist-test-slave-gp6n"
server_key: "ee1d5edf3ef279a2eb30fcd7aab35409"
server_key_iv: "3a0ff046b1decd1f1859f7d66c5673b6"
server_key_version: "encryptionkey@0"
I20260808 18:31:36.120448 14812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260808 18:31:36.122967 14828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:36.124169 14812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:31:36.124305 14812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "08c941ac7e764e27abd46653e2b8c323"
format_stamp: "Formatted at 2026-08-08 18:31:36 on dist-test-slave-gp6n"
server_key: "ee1d5edf3ef279a2eb30fcd7aab35409"
server_key_iv: "3a0ff046b1decd1f1859f7d66c5673b6"
server_key_version: "encryptionkey@0"
I20260808 18:31:36.124416 14812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:36.149606 14812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:36.152443 14812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:36.152645 14812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:36.153232 14812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:31:36.154067 14812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:31:36.154139 14812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:36.154211 14812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:31:36.154264 14812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:36.163796 14812 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:44727
I20260808 18:31:36.163832 14958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:44727 every 8 connection(s)
I20260808 18:31:36.164831 14812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:31:36.168351 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 14812
I20260808 18:31:36.168440 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:31:36.168740 30010 external_mini_cluster.cc:1428] Setting key c43774f514d85388c11ad6fd80997e23
Aug 08 18:31:36 dist-test-slave-gp6n krb5kdc[14551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213896, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260808 18:31:36.179670 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:36.166844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:40699 (local address 127.29.78.190:44257)
0808 18:31:36.166977 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:36.166980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:36.167962 (+ 982us) server_negotiation.cc:407] Connection header received
0808 18:31:36.169007 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:36.169010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:36.169089 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:36.169185 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:36.170693 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.171220 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.172855 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.173251 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.173376 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:36.175960 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:36.175963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:36.175965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:36.175989 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:36.177492 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:36.178026 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:36.178030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:36.178032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:36.178122 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:36.178509 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:36.178512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:36.178514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:36.178704 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:36.178817 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:36.179360 (+ 543us) server_negotiation.cc:299] Negotiation successful
0808 18:31:36.179485 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 18:31:36.180596 14961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:36.167180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.130:40699)
0808 18:31:36.167812 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:36.167854 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:36.168787 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:36.169405 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:36.169414 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:36.169803 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:36.170541 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:36.170562 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.171353 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.171360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:36.172727 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:36.172736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.173407 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.173410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:36.173564 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:36.174201 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:36.174217 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:36.175786 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:36.177634 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:36.177640 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:36.177654 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:36.177919 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:36.178218 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:36.178223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:36.178225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:36.178367 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:36.178816 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:36.178822 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:36.179115 (+ 293us) client_negotiation.cc:769] Sending connection context
0808 18:31:36.179297 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 18:31:36.179616 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":109,"threads_started":1}
I20260808 18:31:36.181766 14959 heartbeater.cc:347] Connected to a master server at 127.29.78.190:44257
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
I20260808 18:31:36.182075 14959 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:36.182651 14959 heartbeater.cc:511] Master 127.29.78.190:44257 requested a full tablet report, sending...
I20260808 18:31:36.183760 14598 ts_manager.cc:194] Registered new tserver with Master: 08c941ac7e764e27abd46653e2b8c323 (127.29.78.130:44727)
I20260808 18:31:36.184402 14598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:40699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260808 18:31:36.198902 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:44257
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:31:36.312602 14964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:36.312878 14964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:36.312948 14964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:36.317090 14964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:36.317178 14964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:36.317265 14964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:31:36.322618 14964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:44257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:36.323766 14964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:36.325060 14964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:36.335534 14972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:36.335580 14964 server_base.cc:1034] running on GCE node
W20260808 18:31:36.335530 14969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:36.335507 14970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:36.336133 14964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:36.336782 14964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:36.337967 14964 hybrid_clock.cc:648] HybridClock initialized: now 1786213896337936 us; error 42 us; skew 500 ppm
Aug 08 18:31:36 dist-test-slave-gp6n krb5kdc[14551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213896, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:36.340723 14964 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:31:36.341616 14964 webserver.cc:460] Webserver started at http://127.29.78.131:33911/ using document root <none> and password file <none>
I20260808 18:31:36.342199 14964 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:36.342259 14964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:36.342558 14964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:31:36.344611 14964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "960ac1fba0aa4862ada0a3652663025e"
format_stamp: "Formatted at 2026-08-08 18:31:36 on dist-test-slave-gp6n"
server_key: "bb83dc82a2cd7050f82ebebc3a124aea"
server_key_iv: "e40ff0f884b62658c7f51d3c80d91481"
server_key_version: "encryptionkey@0"
I20260808 18:31:36.345192 14964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "960ac1fba0aa4862ada0a3652663025e"
format_stamp: "Formatted at 2026-08-08 18:31:36 on dist-test-slave-gp6n"
server_key: "bb83dc82a2cd7050f82ebebc3a124aea"
server_key_iv: "e40ff0f884b62658c7f51d3c80d91481"
server_key_version: "encryptionkey@0"
I20260808 18:31:36.349081 14964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260808 18:31:36.351564 14979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:36.353014 14964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:31:36.353173 14964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "960ac1fba0aa4862ada0a3652663025e"
format_stamp: "Formatted at 2026-08-08 18:31:36 on dist-test-slave-gp6n"
server_key: "bb83dc82a2cd7050f82ebebc3a124aea"
server_key_iv: "e40ff0f884b62658c7f51d3c80d91481"
server_key_version: "encryptionkey@0"
I20260808 18:31:36.353294 14964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:36.382295 14964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:36.385309 14964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:36.385535 14964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:36.386161 14964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:31:36.387058 14964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:31:36.387128 14964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:36.387202 14964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:31:36.387244 14964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:36.396040 14964 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:45375
I20260808 18:31:36.396075 15106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:45375 every 8 connection(s)
I20260808 18:31:36.396942 14964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:31:36.405468 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 14964
I20260808 18:31:36.405613 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:31:36.405962 30010 external_mini_cluster.cc:1428] Setting key 91a9f6a888e75a7ad2049496103860c0
Aug 08 18:31:36 dist-test-slave-gp6n krb5kdc[14551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213896, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:31:36.410817 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:36.398741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:55047 (local address 127.29.78.190:44257)
0808 18:31:36.398901 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:36.398904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:36.399794 (+ 890us) server_negotiation.cc:407] Connection header received
0808 18:31:36.401218 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:36.401221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:36.401281 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:36.401362 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:36.402550 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.403094 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.404234 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.404463 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.404575 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:36.407483 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:36.407487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:36.407490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:36.407517 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:36.408992 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:36.409466 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:36.409469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:36.409470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:36.409556 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:36.409846 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:36.409848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:36.409849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:36.410017 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:36.410095 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:36.410600 (+ 505us) server_negotiation.cc:299] Negotiation successful
0808 18:31:36.410703 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 18:31:36.411659 15110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:36.399142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.131:55047)
0808 18:31:36.399621 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:36.399693 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:36.401011 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:36.401534 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:36.401540 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:36.401894 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:36.402354 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:36.402367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.403223 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.403227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:36.404106 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:36.404115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.404614 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.404618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:36.404786 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:36.405344 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:36.405365 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:36.407300 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:36.409096 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:36.409101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:36.409112 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:36.409382 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:36.409637 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:36.409639 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:36.409640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:36.409769 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:36.410103 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:36.410108 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:36.410326 (+ 218us) client_negotiation.cc:769] Sending connection context
0808 18:31:36.410554 (+ 228us) client_negotiation.cc:240] Negotiation successful
0808 18:31:36.410807 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":120,"threads_started":1}
I20260808 18:31:36.412786 15107 heartbeater.cc:347] Connected to a master server at 127.29.78.190:44257
I20260808 18:31:36.413023 15107 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:36.413494 15107 heartbeater.cc:511] Master 127.29.78.190:44257 requested a full tablet report, sending...
I20260808 18:31:36.414542 14598 ts_manager.cc:194] Registered new tserver with Master: 960ac1fba0aa4862ada0a3652663025e (127.29.78.131:45375)
I20260808 18:31:36.415094 14598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:55047
I20260808 18:31:36.419865 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:31:36.428946 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:36.421841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44930 (local address 127.29.78.190:44257)
0808 18:31:36.421982 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:36.421987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:36.422188 (+ 201us) server_negotiation.cc:407] Connection header received
0808 18:31:36.422282 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:36.422284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:36.422322 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:36.422389 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:36.423093 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.423592 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.424382 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.424594 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.424668 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:36.425980 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:36.425982 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:36.425984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:36.426001 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:36.427384 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:36.427846 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:36.427849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:36.427850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:36.427926 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:36.428149 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:36.428151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:36.428152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:36.428337 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:36.428437 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:36.428605 (+ 168us) server_negotiation.cc:299] Negotiation successful
0808 18:31:36.428755 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:31:36.431891 14598 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:31:36.434988 14598 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:31:36.449988 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:36.443516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44727 (local address 127.0.0.1:59082)
0808 18:31:36.443898 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:36.443950 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:36.444083 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:36.445316 (+ 1233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:36.445322 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:36.445343 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:36.445514 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:36.445523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.447057 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.447061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:36.448466 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:36.448477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.449568 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.449572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:36.449659 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:36.449681 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 18:31:36.449743 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 18:31:36.449818 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":24,"thread_start_us":100,"threads_started":1}
I20260808 18:31:36.450011 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:36.443665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59082 (local address 127.29.78.130:44727)
0808 18:31:36.444853 (+ 1188us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:36.444863 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:36.444883 (+ 20us) server_negotiation.cc:407] Connection header received
0808 18:31:36.444947 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:36.444951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:36.445139 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:36.445304 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:36.445655 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.446884 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.448614 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.449442 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.449575 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:36.449714 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:36.449780 (+ 66us) server_negotiation.cc:299] Negotiation successful
0808 18:31:36.449869 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1093,"thread_start_us":172,"threads_started":1}
I20260808 18:31:36.449848 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:36.444286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:45375 (local address 127.0.0.1:46304)
0808 18:31:36.444730 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:36.444742 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:36.444829 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:36.445502 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:36.445505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:36.445537 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:36.445732 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:36.445741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.447165 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.447168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:36.448188 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:36.448198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.449233 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.449237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:36.449360 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:36.449406 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 18:31:36.449552 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 18:31:36.449685 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":63,"spinlock_wait_cycles":6016,"thread_start_us":96,"threads_started":1}
I20260808 18:31:36.450654 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:36.444451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46304 (local address 127.29.78.131:45375)
0808 18:31:36.444813 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:36.444825 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:36.444851 (+ 26us) server_negotiation.cc:407] Connection header received
0808 18:31:36.444929 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:36.444934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:36.445139 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:36.445304 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:36.445872 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.447012 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.448320 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.449093 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.449214 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:36.449379 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:36.449917 (+ 538us) server_negotiation.cc:299] Negotiation successful
0808 18:31:36.450005 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":105,"threads_started":1}
I20260808 18:31:36.452654 14884 tablet_service.cc:1467] Processing CreateTablet for tablet d2e7365e0ee843adad06b5dc4521cf8b (DEFAULT_TABLE table=test-table [id=8b2d115f211c43a2b45a00e5d0f86701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:31:36.452823 15033 tablet_service.cc:1467] Processing CreateTablet for tablet d2e7365e0ee843adad06b5dc4521cf8b (DEFAULT_TABLE table=test-table [id=8b2d115f211c43a2b45a00e5d0f86701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:31:36.453460 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:36.443805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:44659 (local address 127.0.0.1:51302)
0808 18:31:36.444398 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:36.444412 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:36.444502 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:36.445502 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:36.445505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:36.445536 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:36.445742 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:36.445748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.447583 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.447586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:36.448693 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:36.448701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.449852 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.449857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:36.449952 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:36.449973 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 18:31:36.450060 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 18:31:36.453334 (+ 3274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":75,"spinlock_wait_cycles":5248,"thread_start_us":102,"threads_started":1}
I20260808 18:31:36.453469 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:36.444384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51302 (local address 127.29.78.129:44659)
0808 18:31:36.445080 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:36.445094 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:36.445113 (+ 19us) server_negotiation.cc:407] Connection header received
0808 18:31:36.445174 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:36.445177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:36.445378 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:36.445539 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:36.446501 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.447410 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.448830 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.449566 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.449662 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:36.449791 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:36.453164 (+ 3373us) server_negotiation.cc:299] Negotiation successful
0808 18:31:36.453308 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":123,"threads_started":1}
I20260808 18:31:36.453750 14884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e7365e0ee843adad06b5dc4521cf8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:31:36.453819 15033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e7365e0ee843adad06b5dc4521cf8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:31:36.455947 14737 tablet_service.cc:1467] Processing CreateTablet for tablet d2e7365e0ee843adad06b5dc4521cf8b (DEFAULT_TABLE table=test-table [id=8b2d115f211c43a2b45a00e5d0f86701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:31:36.456866 14737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e7365e0ee843adad06b5dc4521cf8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:31:36.460654 15125 tablet_bootstrap.cc:492] T d2e7365e0ee843adad06b5dc4521cf8b P 08c941ac7e764e27abd46653e2b8c323: Bootstrap starting.
I20260808 18:31:36.463078 15125 tablet_bootstrap.cc:654] T d2e7365e0ee843adad06b5dc4521cf8b P 08c941ac7e764e27abd46653e2b8c323: Neither blocks nor log segments found. Creating new log.
I20260808 18:31:36.463639 15126 tablet_bootstrap.cc:492] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73: Bootstrap starting.
I20260808 18:31:36.464138 15125 log.cc:824] T d2e7365e0ee843adad06b5dc4521cf8b P 08c941ac7e764e27abd46653e2b8c323: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:36.464556 15127 tablet_bootstrap.cc:492] T d2e7365e0ee843adad06b5dc4521cf8b P 960ac1fba0aa4862ada0a3652663025e: Bootstrap starting.
I20260808 18:31:36.466041 15126 tablet_bootstrap.cc:654] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73: Neither blocks nor log segments found. Creating new log.
I20260808 18:31:36.466719 15125 tablet_bootstrap.cc:492] T d2e7365e0ee843adad06b5dc4521cf8b P 08c941ac7e764e27abd46653e2b8c323: No bootstrap required, opened a new log
I20260808 18:31:36.466863 15126 log.cc:824] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:36.466915 15125 ts_tablet_manager.cc:1397] T d2e7365e0ee843adad06b5dc4521cf8b P 08c941ac7e764e27abd46653e2b8c323: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:31:36.466967 15127 tablet_bootstrap.cc:654] T d2e7365e0ee843adad06b5dc4521cf8b P 960ac1fba0aa4862ada0a3652663025e: Neither blocks nor log segments found. Creating new log.
I20260808 18:31:36.467769 15127 log.cc:824] T d2e7365e0ee843adad06b5dc4521cf8b P 960ac1fba0aa4862ada0a3652663025e: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:36.468819 15126 tablet_bootstrap.cc:492] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73: No bootstrap required, opened a new log
I20260808 18:31:36.469020 15126 ts_tablet_manager.cc:1397] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:31:36.469411 15127 tablet_bootstrap.cc:492] T d2e7365e0ee843adad06b5dc4521cf8b P 960ac1fba0aa4862ada0a3652663025e: No bootstrap required, opened a new log
I20260808 18:31:36.469594 15127 ts_tablet_manager.cc:1397] T d2e7365e0ee843adad06b5dc4521cf8b P 960ac1fba0aa4862ada0a3652663025e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 18:31:36.470746 15125 raft_consensus.cc:348] T d2e7365e0ee843adad06b5dc4521cf8b P 08c941ac7e764e27abd46653e2b8c323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:31:36.471042 15125 raft_consensus.cc:374] T d2e7365e0ee843adad06b5dc4521cf8b P 08c941ac7e764e27abd46653e2b8c323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:31:36.471098 15125 raft_consensus.cc:729] T d2e7365e0ee843adad06b5dc4521cf8b P 08c941ac7e764e27abd46653e2b8c323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08c941ac7e764e27abd46653e2b8c323, State: Initialized, Role: FOLLOWER
I20260808 18:31:36.471550 15125 consensus_queue.cc:260] T d2e7365e0ee843adad06b5dc4521cf8b P 08c941ac7e764e27abd46653e2b8c323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:31:36.472568 15125 ts_tablet_manager.cc:1428] T d2e7365e0ee843adad06b5dc4521cf8b P 08c941ac7e764e27abd46653e2b8c323: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:31:36.472908 14959 heartbeater.cc:503] Master 127.29.78.190:44257 was elected leader, sending a full tablet report...
I20260808 18:31:36.472803 15127 raft_consensus.cc:348] T d2e7365e0ee843adad06b5dc4521cf8b P 960ac1fba0aa4862ada0a3652663025e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:31:36.473024 15127 raft_consensus.cc:374] T d2e7365e0ee843adad06b5dc4521cf8b P 960ac1fba0aa4862ada0a3652663025e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:31:36.473119 15127 raft_consensus.cc:729] T d2e7365e0ee843adad06b5dc4521cf8b P 960ac1fba0aa4862ada0a3652663025e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 960ac1fba0aa4862ada0a3652663025e, State: Initialized, Role: FOLLOWER
I20260808 18:31:36.473384 15126 raft_consensus.cc:348] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:31:36.473626 15126 raft_consensus.cc:374] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:31:36.473549 15127 consensus_queue.cc:260] T d2e7365e0ee843adad06b5dc4521cf8b P 960ac1fba0aa4862ada0a3652663025e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:31:36.473699 15126 raft_consensus.cc:729] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d85fdfa3db4e9abda1a406fbe28b73, State: Initialized, Role: FOLLOWER
I20260808 18:31:36.474138 15126 consensus_queue.cc:260] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:31:36.474553 15127 ts_tablet_manager.cc:1428] T d2e7365e0ee843adad06b5dc4521cf8b P 960ac1fba0aa4862ada0a3652663025e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 18:31:36.474897 15107 heartbeater.cc:503] Master 127.29.78.190:44257 was elected leader, sending a full tablet report...
I20260808 18:31:36.475108 15126 ts_tablet_manager.cc:1428] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260808 18:31:36.475287 14807 heartbeater.cc:503] Master 127.29.78.190:44257 was elected leader, sending a full tablet report...
W20260808 18:31:36.647863 15109 tablet.cc:2369] T d2e7365e0ee843adad06b5dc4521cf8b P 960ac1fba0aa4862ada0a3652663025e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:31:36.661782 15133 raft_consensus.cc:482] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:31:36.662014 15133 raft_consensus.cc:504] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:31:36.663177 15133 leader_election.cc:288] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08c941ac7e764e27abd46653e2b8c323 (127.29.78.130:44727), 960ac1fba0aa4862ada0a3652663025e (127.29.78.131:45375)
W20260808 18:31:36.666077 14960 tablet.cc:2369] T d2e7365e0ee843adad06b5dc4521cf8b P 08c941ac7e764e27abd46653e2b8c323: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:31:36.668040 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:36.663488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44727 (local address 127.29.78.129:59007)
0808 18:31:36.663876 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:36.663890 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:36.664038 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:36.664386 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:36.664389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:36.664420 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:36.664587 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:36.664597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.665635 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.665637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:36.666865 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:36.666876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.667640 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.667642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:36.667713 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:36.667733 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 18:31:36.667791 (+ 58us) client_negotiation.cc:240] Negotiation successful
0808 18:31:36.667858 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":163,"threads_started":1}
I20260808 18:31:36.668061 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:36.663564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:59007 (local address 127.29.78.130:44727)
0808 18:31:36.663743 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:36.663748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:36.663982 (+ 234us) server_negotiation.cc:407] Connection header received
0808 18:31:36.664187 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:36.664190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:36.664250 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:36.664371 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:36.664747 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.665529 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.666989 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.667527 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.667646 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:36.667700 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:36.667833 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 18:31:36.667926 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 18:31:36.668365 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:36.664137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:37347 (local address 127.29.78.131:45375)
0808 18:31:36.664287 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:36.664293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:36.664513 (+ 220us) server_negotiation.cc:407] Connection header received
0808 18:31:36.664721 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:36.664726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:36.664777 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:36.664869 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:36.665156 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.665892 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.667223 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.667833 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.667946 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:36.667991 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:36.668215 (+ 224us) server_negotiation.cc:299] Negotiation successful
0808 18:31:36.668278 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 18:31:36.668752 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:36.664010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:45375 (local address 127.29.78.129:37347)
0808 18:31:36.664434 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:36.664447 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:36.664537 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:36.664914 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:36.664917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:36.664935 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:36.665037 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:36.665042 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.666023 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.666026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:36.667107 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:36.667114 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.667951 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.667954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:36.668042 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:36.668073 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 18:31:36.668528 (+ 455us) client_negotiation.cc:240] Negotiation successful
0808 18:31:36.668621 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":118,"threads_started":1}
I20260808 18:31:36.668874 14906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e7365e0ee843adad06b5dc4521cf8b" candidate_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08c941ac7e764e27abd46653e2b8c323" is_pre_election: true
I20260808 18:31:36.669229 14906 raft_consensus.cc:2454] T d2e7365e0ee843adad06b5dc4521cf8b P 08c941ac7e764e27abd46653e2b8c323 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78d85fdfa3db4e9abda1a406fbe28b73 in term 0.
I20260808 18:31:36.669344 15056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e7365e0ee843adad06b5dc4521cf8b" candidate_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "960ac1fba0aa4862ada0a3652663025e" is_pre_election: true
I20260808 18:31:36.669672 15056 raft_consensus.cc:2454] T d2e7365e0ee843adad06b5dc4521cf8b P 960ac1fba0aa4862ada0a3652663025e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78d85fdfa3db4e9abda1a406fbe28b73 in term 0.
I20260808 18:31:36.669746 14690 leader_election.cc:302] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08c941ac7e764e27abd46653e2b8c323, 78d85fdfa3db4e9abda1a406fbe28b73; no voters:
I20260808 18:31:36.670107 15133 raft_consensus.cc:2781] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:31:36.670184 15133 raft_consensus.cc:482] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:31:36.670215 15133 raft_consensus.cc:3037] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:31:36.671327 15133 raft_consensus.cc:504] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:31:36.671674 15133 leader_election.cc:288] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 1 election: Requested vote from peers 08c941ac7e764e27abd46653e2b8c323 (127.29.78.130:44727), 960ac1fba0aa4862ada0a3652663025e (127.29.78.131:45375)
I20260808 18:31:36.672065 15056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e7365e0ee843adad06b5dc4521cf8b" candidate_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "960ac1fba0aa4862ada0a3652663025e"
I20260808 18:31:36.672065 14906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e7365e0ee843adad06b5dc4521cf8b" candidate_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08c941ac7e764e27abd46653e2b8c323"
I20260808 18:31:36.672165 15056 raft_consensus.cc:3037] T d2e7365e0ee843adad06b5dc4521cf8b P 960ac1fba0aa4862ada0a3652663025e [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:31:36.672202 14906 raft_consensus.cc:3037] T d2e7365e0ee843adad06b5dc4521cf8b P 08c941ac7e764e27abd46653e2b8c323 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:31:36.673211 15056 raft_consensus.cc:2454] T d2e7365e0ee843adad06b5dc4521cf8b P 960ac1fba0aa4862ada0a3652663025e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78d85fdfa3db4e9abda1a406fbe28b73 in term 1.
I20260808 18:31:36.673259 14906 raft_consensus.cc:2454] T d2e7365e0ee843adad06b5dc4521cf8b P 08c941ac7e764e27abd46653e2b8c323 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78d85fdfa3db4e9abda1a406fbe28b73 in term 1.
I20260808 18:31:36.673498 14690 leader_election.cc:302] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78d85fdfa3db4e9abda1a406fbe28b73, 960ac1fba0aa4862ada0a3652663025e; no voters:
I20260808 18:31:36.673682 15133 raft_consensus.cc:2781] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:31:36.673910 15133 raft_consensus.cc:686] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 LEADER]: Becoming Leader. State: Replica: 78d85fdfa3db4e9abda1a406fbe28b73, State: Running, Role: LEADER
I20260808 18:31:36.674255 15133 consensus_queue.cc:237] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:31:36.677328 14597 catalog_manager.cc:5495] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78d85fdfa3db4e9abda1a406fbe28b73 (127.29.78.129). New cstate: current_term: 1 leader_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } health_report { overall_health: HEALTHY } } }
W20260808 18:31:36.688766 14808 tablet.cc:2369] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:31:36.740516 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:36.736720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51306 (local address 127.29.78.129:44659)
0808 18:31:36.736874 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:36.736879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:36.736895 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:31:36.737023 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:36.737025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:36.737094 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:36.737219 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:31:36.737493 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.738077 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.738981 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:36.739210 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:36.739322 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:36.739657 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:31:36.740114 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:31:36.740250 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:36.740311 (+ 61us) server_negotiation.cc:299] Negotiation successful
0808 18:31:36.740370 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 18:31:36.746753 14906 raft_consensus.cc:1261] T d2e7365e0ee843adad06b5dc4521cf8b P 08c941ac7e764e27abd46653e2b8c323 [term 1 FOLLOWER]: Refusing update from remote peer 78d85fdfa3db4e9abda1a406fbe28b73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:31:36.746753 15056 raft_consensus.cc:1261] T d2e7365e0ee843adad06b5dc4521cf8b P 960ac1fba0aa4862ada0a3652663025e [term 1 FOLLOWER]: Refusing update from remote peer 78d85fdfa3db4e9abda1a406fbe28b73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:31:36.747574 15133 consensus_queue.cc:1035] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:31:36.747813 15137 consensus_queue.cc:1035] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:31:36.755961 15142 mvcc.cc:205] Tried to move back new op lower bound from 7316332121066532864 to 7316332120780754944. Current Snapshot: MvccSnapshot[applied={T|T < 7316332121066532864}]
I20260808 18:31:36.759279 15143 mvcc.cc:205] Tried to move back new op lower bound from 7316332121066532864 to 7316332120780754944. Current Snapshot: MvccSnapshot[applied={T|T < 7316332121066532864}]
I20260808 18:31:36.763803 14597 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:31:36.764003 14597 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:31:36.766497 14597 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea: Sending DeleteTablet for 3 replicas of tablet d2e7365e0ee843adad06b5dc4521cf8b
I20260808 18:31:36.767369 14884 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e7365e0ee843adad06b5dc4521cf8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:31:36 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:59082
I20260808 18:31:36.767372 14737 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e7365e0ee843adad06b5dc4521cf8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:31:36 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:51302
I20260808 18:31:36.767870 15152 tablet_replica.cc:330] T d2e7365e0ee843adad06b5dc4521cf8b P 08c941ac7e764e27abd46653e2b8c323: stopping tablet replica
I20260808 18:31:36.768106 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 14668
I20260808 18:31:36.768193 15033 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e7365e0ee843adad06b5dc4521cf8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:31:36 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:46304
I20260808 18:31:36.772289 15152 raft_consensus.cc:2229] T d2e7365e0ee843adad06b5dc4521cf8b P 08c941ac7e764e27abd46653e2b8c323 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:31:36.772639 15152 raft_consensus.cc:2258] T d2e7365e0ee843adad06b5dc4521cf8b P 08c941ac7e764e27abd46653e2b8c323 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:31:36.774204 15152 ts_tablet_manager.cc:1905] T d2e7365e0ee843adad06b5dc4521cf8b P 08c941ac7e764e27abd46653e2b8c323: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:31:36.774629 15153 tablet_replica.cc:330] T d2e7365e0ee843adad06b5dc4521cf8b P 960ac1fba0aa4862ada0a3652663025e: stopping tablet replica
I20260808 18:31:36.774976 15153 raft_consensus.cc:2229] T d2e7365e0ee843adad06b5dc4521cf8b P 960ac1fba0aa4862ada0a3652663025e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:31:36.775285 15153 raft_consensus.cc:2258] T d2e7365e0ee843adad06b5dc4521cf8b P 960ac1fba0aa4862ada0a3652663025e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:31:36.776754 15152 ts_tablet_manager.cc:1918] T d2e7365e0ee843adad06b5dc4521cf8b P 08c941ac7e764e27abd46653e2b8c323: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:31:36.776866 15152 log.cc:1196] T d2e7365e0ee843adad06b5dc4521cf8b P 08c941ac7e764e27abd46653e2b8c323: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/d2e7365e0ee843adad06b5dc4521cf8b
I20260808 18:31:36.776867 15153 ts_tablet_manager.cc:1905] T d2e7365e0ee843adad06b5dc4521cf8b P 960ac1fba0aa4862ada0a3652663025e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:31:36.777215 15152 ts_tablet_manager.cc:1939] T d2e7365e0ee843adad06b5dc4521cf8b P 08c941ac7e764e27abd46653e2b8c323: Deleting consensus metadata
I20260808 18:31:36.778342 14584 catalog_manager.cc:4841] TS 08c941ac7e764e27abd46653e2b8c323 (127.29.78.130:44727): tablet d2e7365e0ee843adad06b5dc4521cf8b (table test-table [id=8b2d115f211c43a2b45a00e5d0f86701]) successfully deleted
W20260808 18:31:36.779798 14582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:44659 (error 108)
I20260808 18:31:36.780054 15153 ts_tablet_manager.cc:1918] T d2e7365e0ee843adad06b5dc4521cf8b P 960ac1fba0aa4862ada0a3652663025e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:31:36.780128 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 14812
I20260808 18:31:36.780171 15153 log.cc:1196] T d2e7365e0ee843adad06b5dc4521cf8b P 960ac1fba0aa4862ada0a3652663025e: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/d2e7365e0ee843adad06b5dc4521cf8b
I20260808 18:31:36.780494 15153 ts_tablet_manager.cc:1939] T d2e7365e0ee843adad06b5dc4521cf8b P 960ac1fba0aa4862ada0a3652663025e: Deleting consensus metadata
I20260808 18:31:36.781778 14584 catalog_manager.cc:4841] TS 960ac1fba0aa4862ada0a3652663025e (127.29.78.131:45375): tablet d2e7365e0ee843adad06b5dc4521cf8b (table test-table [id=8b2d115f211c43a2b45a00e5d0f86701]) successfully deleted
I20260808 18:31:36.782859 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:36.780742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:44659 (local address 127.0.0.1:51320)
0808 18:31:36.782671 (+ 1929us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:36.782755 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:44659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1818}
W20260808 18:31:36.783051 14582 catalog_manager.cc:4584] TS 78d85fdfa3db4e9abda1a406fbe28b73 (127.29.78.129:44659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2e7365e0ee843adad06b5dc4521cf8b: Network error: Client connection negotiation failed: client connection to 127.29.78.129:44659: connect: Connection refused (error 111)
I20260808 18:31:36.788062 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 14964
I20260808 18:31:36.796011 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 14565
I20260808 18:31:56.804975 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:44257
--webserver_interface=127.29.78.190
--webserver_port=36637
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:44257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:31:56.918146 15155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:56.918430 15155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:56.918473 15155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:56.922551 15155 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 18:31:56.922600 15155 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 18:31:56.922616 15155 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 18:31:56.922631 15155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:31:56.922670 15155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:56.922705 15155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:56.922736 15155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:31:56.922775 15155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:31:56.927878 15155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:44257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:44257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=36637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:56.928936 15155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:56.930135 15155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:31:56.944159 15155 server_base.cc:1034] running on GCE node
W20260808 18:31:56.944185 15164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:56.944162 15161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:56.944425 15162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:56.945086 15155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:56.946250 15155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:56.947456 15155 hybrid_clock.cc:648] HybridClock initialized: now 1786213916947438 us; error 29 us; skew 500 ppm
Aug 08 18:31:56 dist-test-slave-gp6n krb5kdc[14551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213916, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:56.950637 15155 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:31:56.951638 15155 webserver.cc:460] Webserver started at http://127.29.78.190:36637/ using document root <none> and password file <none>
I20260808 18:31:56.952229 15155 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:56.952306 15155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:56.956343 15155 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:31:56.958657 15172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:56.959966 15155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:31:56.960100 15155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "047242e7bd634821bc96757a00bd1dea"
format_stamp: "Formatted at 2026-08-08 18:31:35 on dist-test-slave-gp6n"
server_key: "e3113693b33e4a523163d8e588301262"
server_key_iv: "33ff67cdf99bc2b4f26fc36f30049bc1"
server_key_version: "encryptionkey@0"
I20260808 18:31:56.960705 15155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:56.978386 15155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:56.981352 15155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:56.981557 15155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:56.989095 15155 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:44257
I20260808 18:31:56.989404 15229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:44257 every 8 connection(s)
I20260808 18:31:56.990310 15155 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:31:56.992358 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 15155
I20260808 18:31:56.992812 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:44659
--local_ip_for_outbound_sockets=127.29.78.129
--tserver_master_addrs=127.29.78.190:44257
--webserver_port=37093
--webserver_interface=127.29.78.129
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:31:56.994907 15230 sys_catalog.cc:263] Verifying existing consensus state
I20260808 18:31:56.996623 15230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea: Bootstrap starting.
I20260808 18:31:57.014729 15230 log.cc:824] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:57.024034 15230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:31:57.024446 15230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea: Bootstrap complete.
I20260808 18:31:57.028687 15230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047242e7bd634821bc96757a00bd1dea" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 44257 } }
I20260808 18:31:57.029064 15230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 047242e7bd634821bc96757a00bd1dea, State: Initialized, Role: FOLLOWER
I20260808 18:31:57.029547 15230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047242e7bd634821bc96757a00bd1dea" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 44257 } }
I20260808 18:31:57.029659 15230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:31:57.029745 15230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:31:57.029850 15230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 1 FOLLOWER]: Advancing to term 2
I20260808 18:31:57.031780 15230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047242e7bd634821bc96757a00bd1dea" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 44257 } }
I20260808 18:31:57.032105 15230 leader_election.cc:302] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 047242e7bd634821bc96757a00bd1dea; no voters:
I20260808 18:31:57.032424 15230 leader_election.cc:288] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 18:31:57.032516 15234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 2 FOLLOWER]: Leader election won for term 2
I20260808 18:31:57.032769 15234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 2 LEADER]: Becoming Leader. State: Replica: 047242e7bd634821bc96757a00bd1dea, State: Running, Role: LEADER
I20260808 18:31:57.033133 15234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047242e7bd634821bc96757a00bd1dea" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 44257 } }
I20260808 18:31:57.033710 15230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:31:57.034556 15235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "047242e7bd634821bc96757a00bd1dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047242e7bd634821bc96757a00bd1dea" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 44257 } } }
I20260808 18:31:57.034693 15235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [sys.catalog]: This master's current role is: LEADER
I20260808 18:31:57.035013 15236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 047242e7bd634821bc96757a00bd1dea. Latest consensus state: current_term: 2 leader_uuid: "047242e7bd634821bc96757a00bd1dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047242e7bd634821bc96757a00bd1dea" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 44257 } } }
I20260808 18:31:57.035152 15236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [sys.catalog]: This master's current role is: LEADER
I20260808 18:31:57.035432 15244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:31:57.040393 15244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:31:57.041311 15244 catalog_manager.cc:1259] Loaded cluster ID: b2d9f365084e4c51b109660263e5577a
I20260808 18:31:57.041374 15244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:31:57.043777 15244 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:31:57.044464 15244 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea: Loaded TSK: 0
I20260808 18:31:57.060119 15244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea: Generated new TSK 1
I20260808 18:31:57.060372 15244 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 18:31:57.115723 15232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:57.115978 15232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:57.116020 15232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:57.120178 15232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:57.120240 15232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:57.120325 15232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:31:57.125612 15232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:44659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=37093
--tserver_master_addrs=127.29.78.190:44257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:57.126648 15232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:57.127645 15232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:57.138564 15261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:57.138579 15260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:57.138677 15263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:57.139096 15232 server_base.cc:1034] running on GCE node
I20260808 18:31:57.139549 15232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:57.140249 15232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:57.141424 15232 hybrid_clock.cc:648] HybridClock initialized: now 1786213917141397 us; error 43 us; skew 500 ppm
Aug 08 18:31:57 dist-test-slave-gp6n krb5kdc[14551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213917, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:57.151417 15232 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:31:57.152498 15232 webserver.cc:460] Webserver started at http://127.29.78.129:37093/ using document root <none> and password file <none>
I20260808 18:31:57.153071 15232 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:57.153151 15232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:57.157163 15232 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:31:57.159415 15270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:57.160663 15232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:31:57.160773 15232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "78d85fdfa3db4e9abda1a406fbe28b73"
format_stamp: "Formatted at 2026-08-08 18:31:35 on dist-test-slave-gp6n"
server_key: "92e8ff621682108b6b24496ea7cbc2a1"
server_key_iv: "193b142c1770fc53af6b725ce3b52d44"
server_key_version: "encryptionkey@0"
I20260808 18:31:57.161293 15232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:57.178895 15232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:57.181710 15232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:57.181896 15232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:57.182520 15232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:31:57.183554 15279 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 18:31:57.187690 15232 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 18:31:57.187757 15232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260808 18:31:57.187812 15232 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 18:31:57.189893 15232 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 18:31:57.189946 15232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 18:31:57.190237 15279 tablet_bootstrap.cc:492] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73: Bootstrap starting.
I20260808 18:31:57.201354 15232 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:44659
I20260808 18:31:57.201426 15393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:44659 every 8 connection(s)
I20260808 18:31:57.202462 15232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:31:57.210271 15279 log.cc:824] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:57.211004 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 15232
I20260808 18:31:57.211506 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:44727
--local_ip_for_outbound_sockets=127.29.78.130
--tserver_master_addrs=127.29.78.190:44257
--webserver_port=34837
--webserver_interface=127.29.78.130
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 18:31:57 dist-test-slave-gp6n krb5kdc[14551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213917, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:31:57.219674 15279 tablet_bootstrap.cc:492] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:31:57.220301 15279 tablet_bootstrap.cc:492] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73: Bootstrap complete.
I20260808 18:31:57.221005 15398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:57.204697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.129:35615)
0808 18:31:57.205391 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:57.205450 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:57.206325 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:57.208087 (+ 1762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:57.208095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:57.208496 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:57.208963 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:57.208978 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.211008 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.211011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:57.212320 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:57.212331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.213534 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.213539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:57.213705 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:57.214319 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:57.214336 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:57.215838 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:57.218062 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:57.218068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:57.218080 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:57.218344 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:57.218733 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:57.218736 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:57.218738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:57.218876 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:57.219429 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:57.219435 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:57.219681 (+ 246us) client_negotiation.cc:769] Sending connection context
0808 18:31:57.219868 (+ 187us) client_negotiation.cc:240] Negotiation successful
0808 18:31:57.220141 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":159,"threads_started":1}
I20260808 18:31:57.222124 15279 ts_tablet_manager.cc:1397] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.011s
I20260808 18:31:57.222683 15397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:57.204797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:35615 (local address 127.29.78.190:44257)
0808 18:31:57.205265 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:57.205286 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:57.205594 (+ 308us) server_negotiation.cc:407] Connection header received
0808 18:31:57.206987 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:57.207004 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:57.207528 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:57.208038 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:57.209150 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.210831 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.212462 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.213239 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.213430 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:57.216132 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:57.216139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:57.216157 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:57.216187 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:57.217918 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:57.218495 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:57.218501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:57.218505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:57.218626 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:57.218977 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:57.218984 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:57.218986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:57.219313 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:57.219459 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:57.221671 (+ 2212us) server_negotiation.cc:299] Negotiation successful
0808 18:31:57.221909 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":145,"threads_started":1}
I20260808 18:31:57.224854 15395 heartbeater.cc:347] Connected to a master server at 127.29.78.190:44257
I20260808 18:31:57.225265 15395 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:57.225788 15395 heartbeater.cc:511] Master 127.29.78.190:44257 requested a full tablet report, sending...
I20260808 18:31:57.225988 15279 raft_consensus.cc:348] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:31:57.226518 15279 raft_consensus.cc:729] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d85fdfa3db4e9abda1a406fbe28b73, State: Initialized, Role: FOLLOWER
I20260808 18:31:57.227012 15279 consensus_queue.cc:260] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:31:57.228005 15279 ts_tablet_manager.cc:1428] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 18:31:57.229746 15190 ts_manager.cc:194] Registered new tserver with Master: 78d85fdfa3db4e9abda1a406fbe28b73 (127.29.78.129:44659)
I20260808 18:31:57.233134 15190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:35615
I20260808 18:31:57.235922 15395 heartbeater.cc:503] Master 127.29.78.190:44257 was elected leader, sending a full tablet report...
I20260808 18:31:57.238971 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:57.232457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:44659 (local address 127.0.0.1:35614)
0808 18:31:57.234147 (+ 1690us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:57.234192 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:57.234327 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:57.235247 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:57.235253 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:57.235279 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:57.235404 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:57.235410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.236587 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.236590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:57.237681 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:57.237691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.238443 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.238447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:57.238556 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:57.238596 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 18:31:57.238728 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 18:31:57.238847 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1522,"thread_start_us":107,"threads_started":1}
I20260808 18:31:57.238972 15404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:57.233160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35614 (local address 127.29.78.129:44659)
0808 18:31:57.234034 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:57.234047 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:57.234283 (+ 236us) server_negotiation.cc:407] Connection header received
0808 18:31:57.234499 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:57.234505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:57.235051 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:57.235269 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:57.235518 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.236468 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.237791 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.238291 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.238433 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:57.238559 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:57.238760 (+ 201us) server_negotiation.cc:299] Negotiation successful
0808 18:31:57.238843 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":114,"threads_started":1}
I20260808 18:31:57.239866 15322 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e7365e0ee843adad06b5dc4521cf8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:31:36 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:35614
I20260808 18:31:57.240614 15405 tablet_replica.cc:330] stopping tablet replica
I20260808 18:31:57.240883 15405 raft_consensus.cc:2229] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:31:57.241106 15405 raft_consensus.cc:2258] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:31:57.241963 15405 ts_tablet_manager.cc:1905] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:31:57.245510 15405 ts_tablet_manager.cc:1918] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:31:57.245587 15405 log.cc:1196] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/wal/wals/d2e7365e0ee843adad06b5dc4521cf8b
I20260808 18:31:57.245862 15405 ts_tablet_manager.cc:1939] T d2e7365e0ee843adad06b5dc4521cf8b P 78d85fdfa3db4e9abda1a406fbe28b73: Deleting consensus metadata
I20260808 18:31:57.247412 15174 catalog_manager.cc:4841] TS 78d85fdfa3db4e9abda1a406fbe28b73 (127.29.78.129:44659): tablet d2e7365e0ee843adad06b5dc4521cf8b (table test-table [id=8b2d115f211c43a2b45a00e5d0f86701]) successfully deleted
W20260808 18:31:57.331622 15400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:57.331916 15400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:57.331986 15400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:57.336251 15400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:57.336333 15400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:57.336426 15400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:31:57.341562 15400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:44727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=34837
--tserver_master_addrs=127.29.78.190:44257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:57.342635 15400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:57.343628 15400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:57.353216 15411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:57.353308 15414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:57.353212 15412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:57.353274 15400 server_base.cc:1034] running on GCE node
I20260808 18:31:57.353777 15400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:57.354357 15400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:57.355557 15400 hybrid_clock.cc:648] HybridClock initialized: now 1786213917355538 us; error 32 us; skew 500 ppm
Aug 08 18:31:57 dist-test-slave-gp6n krb5kdc[14551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213917, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:57.358225 15400 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:31:57.359210 15400 webserver.cc:460] Webserver started at http://127.29.78.130:34837/ using document root <none> and password file <none>
I20260808 18:31:57.359745 15400 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:57.359838 15400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:57.363583 15400 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260808 18:31:57.366220 15423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:57.367528 15400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 18:31:57.367654 15400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "08c941ac7e764e27abd46653e2b8c323"
format_stamp: "Formatted at 2026-08-08 18:31:36 on dist-test-slave-gp6n"
server_key: "ee1d5edf3ef279a2eb30fcd7aab35409"
server_key_iv: "3a0ff046b1decd1f1859f7d66c5673b6"
server_key_version: "encryptionkey@0"
I20260808 18:31:57.368182 15400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:57.384686 15400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:57.387825 15400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:57.388026 15400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:57.388633 15400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:31:57.389493 15400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:31:57.389540 15400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:57.389601 15400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:31:57.389633 15400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:57.398993 15400 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:44727
I20260808 18:31:57.399013 15547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:44727 every 8 connection(s)
I20260808 18:31:57.399994 15400 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
Aug 08 18:31:57 dist-test-slave-gp6n krb5kdc[14551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213917, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:31:57.409637 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 15400
I20260808 18:31:57.410068 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:45375
--local_ip_for_outbound_sockets=127.29.78.131
--tserver_master_addrs=127.29.78.190:44257
--webserver_port=33911
--webserver_interface=127.29.78.131
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:31:57.414490 15397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:57.401856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:59167 (local address 127.29.78.190:44257)
0808 18:31:57.402008 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:57.402012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:57.402878 (+ 866us) server_negotiation.cc:407] Connection header received
0808 18:31:57.404196 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:57.404199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:57.404261 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:57.404360 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:57.405525 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.406059 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.407264 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.407500 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.407609 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:57.410196 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:57.410200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:57.410202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:57.410238 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:57.411971 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:57.412639 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:57.412642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:57.412643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:57.412717 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:57.413113 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:57.413115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:57.413116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:57.413285 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:57.413360 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:57.413992 (+ 632us) server_negotiation.cc:299] Negotiation successful
0808 18:31:57.414183 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 18:31:57.415115 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:57.402248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.130:59167)
0808 18:31:57.402721 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:57.402765 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:57.403976 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:57.404538 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:57.404545 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:57.404910 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:57.405370 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:57.405383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.406182 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.406186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:57.407136 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:57.407145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.407650 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.407654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:57.407815 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:57.408401 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:57.408417 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:57.410025 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:57.412127 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:57.412135 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:57.412152 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:57.412540 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:57.412806 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:57.412810 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:57.412814 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:57.413017 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:57.413386 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:57.413392 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:57.413706 (+ 314us) client_negotiation.cc:769] Sending connection context
0808 18:31:57.413917 (+ 211us) client_negotiation.cc:240] Negotiation successful
0808 18:31:57.414172 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":134,"threads_started":1}
I20260808 18:31:57.416460 15548 heartbeater.cc:347] Connected to a master server at 127.29.78.190:44257
I20260808 18:31:57.416734 15548 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:57.417268 15548 heartbeater.cc:511] Master 127.29.78.190:44257 requested a full tablet report, sending...
I20260808 18:31:57.418485 15190 ts_manager.cc:194] Registered new tserver with Master: 08c941ac7e764e27abd46653e2b8c323 (127.29.78.130:44727)
I20260808 18:31:57.419126 15190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:59167
W20260808 18:31:57.522783 15551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:31:57.523051 15551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:31:57.523136 15551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:31:57.527938 15551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:31:57.528015 15551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:31:57.528147 15551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:31:57.534194 15551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:45375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=33911
--tserver_master_addrs=127.29.78.190:44257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:31:57.535472 15551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:31:57.536623 15551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:31:57.546226 15557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:57.546216 15556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:31:57.546216 15559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:31:57.546561 15551 server_base.cc:1034] running on GCE node
I20260808 18:31:57.546885 15551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:31:57.547508 15551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:31:57.548660 15551 hybrid_clock.cc:648] HybridClock initialized: now 1786213917548633 us; error 41 us; skew 500 ppm
Aug 08 18:31:57 dist-test-slave-gp6n krb5kdc[14551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213917, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:31:57.551390 15551 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:31:57.552181 15551 webserver.cc:460] Webserver started at http://127.29.78.131:33911/ using document root <none> and password file <none>
I20260808 18:31:57.552739 15551 fs_manager.cc:359] Metadata directory not provided
I20260808 18:31:57.552845 15551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:31:57.556511 15551 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:31:57.558568 15567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:57.559777 15551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:31:57.559896 15551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "960ac1fba0aa4862ada0a3652663025e"
format_stamp: "Formatted at 2026-08-08 18:31:36 on dist-test-slave-gp6n"
server_key: "bb83dc82a2cd7050f82ebebc3a124aea"
server_key_iv: "e40ff0f884b62658c7f51d3c80d91481"
server_key_version: "encryptionkey@0"
I20260808 18:31:57.560431 15551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:31:57.569998 15551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:31:57.573443 15551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:31:57.573642 15551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:31:57.574226 15551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:31:57.575080 15551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:31:57.575147 15551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:57.575217 15551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:31:57.575261 15551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:31:57.583873 15551 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:45375
I20260808 18:31:57.583902 15691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:45375 every 8 connection(s)
I20260808 18:31:57.584844 15551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:31:57.588158 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 15551
Aug 08 18:31:57 dist-test-slave-gp6n krb5kdc[14551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213917, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:31:57.596836 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:57.588819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37612 (local address 127.29.78.190:44257)
0808 18:31:57.589058 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:57.589063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:57.589216 (+ 153us) server_negotiation.cc:407] Connection header received
0808 18:31:57.589384 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:57.589387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:57.589430 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:57.589526 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:57.590251 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.590851 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.591710 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.591935 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.592042 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:57.593544 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:57.593548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:57.593550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:57.593578 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:57.595246 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:57.595644 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:57.595649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:57.595650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:57.595734 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:57.596012 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:57.596016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:57.596017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:57.596191 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:57.596415 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:57.596544 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 18:31:57.596659 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":97,"threads_started":1}
I20260808 18:31:57.598728 15397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:57.586632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:49253 (local address 127.29.78.190:44257)
0808 18:31:57.586789 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:57.586792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:57.587605 (+ 813us) server_negotiation.cc:407] Connection header received
0808 18:31:57.589171 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:57.589175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:57.589233 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:57.589351 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:57.590516 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.591008 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.592118 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.592330 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.592434 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:57.595009 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:57.595012 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:57.595014 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:57.595046 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:57.596427 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:57.597068 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:57.597072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:57.597073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:57.597154 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:57.597562 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:57.597565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:57.597566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:57.597780 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:57.597873 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:57.598463 (+ 590us) server_negotiation.cc:299] Negotiation successful
0808 18:31:57.598578 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 18:31:57.599560 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:57.587017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.131:49253)
0808 18:31:57.587448 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:57.587496 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:57.588926 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:57.589528 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:57.589535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:57.589876 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:31:57.590326 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:57.590338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.591128 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.591132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:57.591998 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:57.592007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.592473 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.592477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:57.592638 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:57.593159 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:31:57.593174 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:31:57.594849 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:31:57.596560 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:57.596565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:57.596578 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:57.596912 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:57.597248 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:31:57.597251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:31:57.597253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:31:57.597445 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:31:57.597893 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:31:57.597899 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:31:57.598182 (+ 283us) client_negotiation.cc:769] Sending connection context
0808 18:31:57.598373 (+ 191us) client_negotiation.cc:240] Negotiation successful
0808 18:31:57.598738 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":116,"threads_started":1}
I20260808 18:31:57.600597 15692 heartbeater.cc:347] Connected to a master server at 127.29.78.190:44257
I20260808 18:31:57.600785 15692 heartbeater.cc:464] Registering TS with master...
I20260808 18:31:57.601202 15692 heartbeater.cc:511] Master 127.29.78.190:44257 requested a full tablet report, sending...
I20260808 18:31:57.602154 15190 ts_manager.cc:194] Registered new tserver with Master: 960ac1fba0aa4862ada0a3652663025e (127.29.78.131:45375)
I20260808 18:31:57.602840 15190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:49253
I20260808 18:31:57.610276 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:31:57.613837 15397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:57.610894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37622 (local address 127.29.78.190:44257)
0808 18:31:57.611010 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:57.611018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:57.611158 (+ 140us) server_negotiation.cc:407] Connection header received
0808 18:31:57.611295 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:57.611297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:57.611382 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:57.611458 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:31:57.611677 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.612155 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.612945 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.613146 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.613219 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:57.613455 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:31:57.613571 (+ 116us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0808 18:31:57.613752 (+ 181us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37622: token expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260808 18:31:57.613909 15397 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37622: token expired
W20260808 18:31:57.614238 15111 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.29.78.190:44257)
I20260808 18:31:57.620657 15397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:57.614600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37638 (local address 127.29.78.190:44257)
0808 18:31:57.614691 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:57.614695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:57.614724 (+ 29us) server_negotiation.cc:407] Connection header received
0808 18:31:57.614806 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:57.614808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:57.614838 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:57.614897 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:31:57.615434 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.615864 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.616554 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.616740 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.616808 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:57.618026 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:31:57.618028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:31:57.618029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:31:57.618046 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:31:57.619275 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:57.619643 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:57.619645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:57.619646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:57.619713 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:31:57.619993 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:31:57.619995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:31:57.619996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:31:57.620171 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:31:57.620270 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:57.620478 (+ 208us) server_negotiation.cc:299] Negotiation successful
0808 18:31:57.620560 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 18:31:57.622941 15190 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:31:57.626072 15190 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:31:57.633430 15322 tablet_service.cc:1467] Processing CreateTablet for tablet 14728559464c4d8ebdd0f83df08a8924 (DEFAULT_TABLE table=test-table [id=eb07d04285944b9b856431fb833be5da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:31:57.634137 15322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14728559464c4d8ebdd0f83df08a8924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:31:57.637033 15279 tablet_bootstrap.cc:492] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73: Bootstrap starting.
I20260808 18:31:57.637723 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:57.632402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:45375 (local address 127.0.0.1:50976)
0808 18:31:57.632522 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:57.632537 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:57.632615 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:57.633767 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:57.633770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:57.633782 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:57.633872 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:57.633879 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.635427 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.635431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:57.636369 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:57.636379 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.637321 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.637325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:57.637415 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:57.637441 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 18:31:57.637506 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 18:31:57.637589 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":36}
I20260808 18:31:57.637876 15279 tablet_bootstrap.cc:654] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73: Neither blocks nor log segments found. Creating new log.
I20260808 18:31:57.638458 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:57.632645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44727 (local address 127.0.0.1:54058)
0808 18:31:57.632975 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:57.632990 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:57.633076 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:57.633720 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:57.633726 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:57.633751 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:57.634081 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:57.634090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.635578 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.635581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:57.636754 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:57.636766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.638071 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.638074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:57.638150 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:57.638168 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 18:31:57.638225 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 18:31:57.638298 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":176,"thread_start_us":111,"threads_started":1}
I20260808 18:31:57.638881 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:57.632827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50976 (local address 127.29.78.131:45375)
0808 18:31:57.633306 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:57.633317 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:57.633338 (+ 21us) server_negotiation.cc:407] Connection header received
0808 18:31:57.633408 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:57.633413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:57.633604 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:57.633804 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:57.634141 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.635319 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.636494 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.637189 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.638313 (+ 1124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:57.638577 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:57.638631 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 18:31:57.638723 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":174,"threads_started":1}
I20260808 18:31:57.639110 15279 tablet_bootstrap.cc:492] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73: No bootstrap required, opened a new log
I20260808 18:31:57.639144 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:57.632821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54058 (local address 127.29.78.130:44727)
0808 18:31:57.633289 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:57.633303 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:57.633321 (+ 18us) server_negotiation.cc:407] Connection header received
0808 18:31:57.633383 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:57.633387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:57.633557 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:57.633859 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:57.634178 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.635454 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.636891 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.637924 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.638728 (+ 804us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:57.638848 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:57.638888 (+ 40us) server_negotiation.cc:299] Negotiation successful
0808 18:31:57.638998 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":119,"threads_started":1}
I20260808 18:31:57.639202 15279 ts_tablet_manager.cc:1397] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260808 18:31:57.639811 15279 raft_consensus.cc:348] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:31:57.639935 15279 raft_consensus.cc:374] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:31:57.639991 15279 raft_consensus.cc:729] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d85fdfa3db4e9abda1a406fbe28b73, State: Initialized, Role: FOLLOWER
I20260808 18:31:57.640123 15279 consensus_queue.cc:260] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:31:57.640380 15279 ts_tablet_manager.cc:1428] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 18:31:57.642088 15622 tablet_service.cc:1467] Processing CreateTablet for tablet 14728559464c4d8ebdd0f83df08a8924 (DEFAULT_TABLE table=test-table [id=eb07d04285944b9b856431fb833be5da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:31:57.643226 15622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14728559464c4d8ebdd0f83df08a8924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:31:57.648135 15475 tablet_service.cc:1467] Processing CreateTablet for tablet 14728559464c4d8ebdd0f83df08a8924 (DEFAULT_TABLE table=test-table [id=eb07d04285944b9b856431fb833be5da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:31:57.648985 15475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14728559464c4d8ebdd0f83df08a8924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:31:57.649598 15705 tablet_bootstrap.cc:492] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e: Bootstrap starting.
I20260808 18:31:57.652027 15705 tablet_bootstrap.cc:654] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e: Neither blocks nor log segments found. Creating new log.
I20260808 18:31:57.652743 15705 log.cc:824] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:57.654495 15705 tablet_bootstrap.cc:492] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e: No bootstrap required, opened a new log
I20260808 18:31:57.654680 15705 ts_tablet_manager.cc:1397] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260808 18:31:57.655381 15707 tablet_bootstrap.cc:492] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323: Bootstrap starting.
I20260808 18:31:57.658245 15707 tablet_bootstrap.cc:654] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323: Neither blocks nor log segments found. Creating new log.
I20260808 18:31:57.658849 15705 raft_consensus.cc:348] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:31:57.659143 15705 raft_consensus.cc:374] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:31:57.659138 15707 log.cc:824] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323: Log is configured to *not* fsync() on all Append() calls
I20260808 18:31:57.659206 15705 raft_consensus.cc:729] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 960ac1fba0aa4862ada0a3652663025e, State: Initialized, Role: FOLLOWER
I20260808 18:31:57.659649 15705 consensus_queue.cc:260] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:31:57.660264 15692 heartbeater.cc:503] Master 127.29.78.190:44257 was elected leader, sending a full tablet report...
I20260808 18:31:57.660553 15705 ts_tablet_manager.cc:1428] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:31:57.661056 15707 tablet_bootstrap.cc:492] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323: No bootstrap required, opened a new log
I20260808 18:31:57.661233 15707 ts_tablet_manager.cc:1397] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:31:57.664485 15707 raft_consensus.cc:348] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:31:57.664743 15707 raft_consensus.cc:374] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:31:57.664805 15707 raft_consensus.cc:729] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08c941ac7e764e27abd46653e2b8c323, State: Initialized, Role: FOLLOWER
I20260808 18:31:57.665200 15707 consensus_queue.cc:260] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:31:57.665759 15548 heartbeater.cc:503] Master 127.29.78.190:44257 was elected leader, sending a full tablet report...
I20260808 18:31:57.666069 15707 ts_tablet_manager.cc:1428] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260808 18:31:57.703747 15396 tablet.cc:2369] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 18:31:57.835863 15693 tablet.cc:2369] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:31:57.838361 15710 raft_consensus.cc:482] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:31:57.838605 15710 raft_consensus.cc:504] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:31:57.839818 15710 leader_election.cc:288] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08c941ac7e764e27abd46653e2b8c323 (127.29.78.130:44727), 78d85fdfa3db4e9abda1a406fbe28b73 (127.29.78.129:44659)
I20260808 18:31:57.844725 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:57.840120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:44659 (local address 127.29.78.131:36325)
0808 18:31:57.840580 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:57.840595 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:57.840670 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:57.841042 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:57.841045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:57.841059 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:57.841175 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:57.841180 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.841923 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.841928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:57.843239 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:57.843252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.844298 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.844300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:57.844383 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:57.844399 (+ 16us) client_negotiation.cc:769] Sending connection context
0808 18:31:57.844501 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 18:31:57.844579 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":1,"thread_start_us":160,"threads_started":1}
I20260808 18:31:57.844712 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:57.840244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:38177 (local address 127.29.78.130:44727)
0808 18:31:57.840682 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:57.840688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:57.840706 (+ 18us) server_negotiation.cc:407] Connection header received
0808 18:31:57.840765 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:57.840768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:57.840818 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:57.840905 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:57.841178 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.842056 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.843405 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.844033 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.844162 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:57.844215 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:57.844510 (+ 295us) server_negotiation.cc:299] Negotiation successful
0808 18:31:57.844585 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348}
I20260808 18:31:57.844744 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:57.840027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44727 (local address 127.29.78.131:38177)
0808 18:31:57.840383 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:57.840401 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:31:57.840502 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:31:57.840915 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:31:57.840918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:31:57.840933 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:31:57.841047 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:57.841054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.842191 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.842194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:57.843271 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:31:57.843281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.844167 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.844171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:31:57.844329 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:57.844372 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 18:31:57.844468 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 18:31:57.844553 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":1}
I20260808 18:31:57.845201 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:57.840236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:36325 (local address 127.29.78.129:44659)
0808 18:31:57.840687 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:57.840695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:57.840714 (+ 19us) server_negotiation.cc:407] Connection header received
0808 18:31:57.840778 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:57.840782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:57.840844 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:57.840949 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:31:57.841297 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.841803 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.843400 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.844033 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.844162 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:57.844215 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:57.844892 (+ 677us) server_negotiation.cc:299] Negotiation successful
0808 18:31:57.845004 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":96,"threads_started":1}
I20260808 18:31:57.845575 15497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14728559464c4d8ebdd0f83df08a8924" candidate_uuid: "960ac1fba0aa4862ada0a3652663025e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08c941ac7e764e27abd46653e2b8c323" is_pre_election: true
I20260808 18:31:57.845712 15345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14728559464c4d8ebdd0f83df08a8924" candidate_uuid: "960ac1fba0aa4862ada0a3652663025e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" is_pre_election: true
I20260808 18:31:57.845928 15497 raft_consensus.cc:2454] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 960ac1fba0aa4862ada0a3652663025e in term 0.
I20260808 18:31:57.846021 15345 raft_consensus.cc:2454] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 960ac1fba0aa4862ada0a3652663025e in term 0.
I20260808 18:31:57.846524 15569 leader_election.cc:302] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08c941ac7e764e27abd46653e2b8c323, 78d85fdfa3db4e9abda1a406fbe28b73, 960ac1fba0aa4862ada0a3652663025e; no voters:
I20260808 18:31:57.846792 15710 raft_consensus.cc:2781] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:31:57.846880 15710 raft_consensus.cc:482] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:31:57.846931 15710 raft_consensus.cc:3037] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:31:57.847991 15710 raft_consensus.cc:504] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:31:57.848361 15710 leader_election.cc:288] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e [CANDIDATE]: Term 1 election: Requested vote from peers 08c941ac7e764e27abd46653e2b8c323 (127.29.78.130:44727), 78d85fdfa3db4e9abda1a406fbe28b73 (127.29.78.129:44659)
I20260808 18:31:57.848724 15497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14728559464c4d8ebdd0f83df08a8924" candidate_uuid: "960ac1fba0aa4862ada0a3652663025e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08c941ac7e764e27abd46653e2b8c323"
I20260808 18:31:57.848765 15345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14728559464c4d8ebdd0f83df08a8924" candidate_uuid: "960ac1fba0aa4862ada0a3652663025e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78d85fdfa3db4e9abda1a406fbe28b73"
I20260808 18:31:57.848852 15497 raft_consensus.cc:3037] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:31:57.848908 15345 raft_consensus.cc:3037] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:31:57.849907 15497 raft_consensus.cc:2454] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 960ac1fba0aa4862ada0a3652663025e in term 1.
I20260808 18:31:57.850227 15572 leader_election.cc:302] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08c941ac7e764e27abd46653e2b8c323, 960ac1fba0aa4862ada0a3652663025e; no voters:
I20260808 18:31:57.850440 15710 raft_consensus.cc:2781] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:31:57.850427 15345 raft_consensus.cc:2454] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 960ac1fba0aa4862ada0a3652663025e in term 1.
I20260808 18:31:57.850662 15710 raft_consensus.cc:686] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e [term 1 LEADER]: Becoming Leader. State: Replica: 960ac1fba0aa4862ada0a3652663025e, State: Running, Role: LEADER
I20260808 18:31:57.851014 15710 consensus_queue.cc:237] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:31:57.854271 15190 catalog_manager.cc:5495] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e reported cstate change: term changed from 0 to 1, leader changed from <none> to 960ac1fba0aa4862ada0a3652663025e (127.29.78.131). New cstate: current_term: 1 leader_uuid: "960ac1fba0aa4862ada0a3652663025e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } health_report { overall_health: UNKNOWN } } }
I20260808 18:31:57.873344 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:57.869135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50980 (local address 127.29.78.131:45375)
0808 18:31:57.869254 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 18:31:57.869264 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 18:31:57.869278 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:31:57.869446 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:31:57.869451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:31:57.869528 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:31:57.869637 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:31:57.869821 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.870702 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.871568 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:31:57.871984 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:31:57.872097 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:31:57.872348 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:31:57.872921 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:31:57.873048 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 18:31:57.873091 (+ 43us) server_negotiation.cc:299] Negotiation successful
0808 18:31:57.873156 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 18:31:57.879580 15497 raft_consensus.cc:1261] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323 [term 1 FOLLOWER]: Refusing update from remote peer 960ac1fba0aa4862ada0a3652663025e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:31:57.879577 15345 raft_consensus.cc:1261] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Refusing update from remote peer 960ac1fba0aa4862ada0a3652663025e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:31:57.880312 15710 consensus_queue.cc:1035] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e [LEADER]: Connected to new peer: Peer: permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:31:57.880513 15714 consensus_queue.cc:1035] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e [LEADER]: Connected to new peer: Peer: permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:31:57.888042 15717 mvcc.cc:205] Tried to move back new op lower bound from 7316332207626883072 to 7316332207520456704. Current Snapshot: MvccSnapshot[applied={T|T < 7316332207626883072}]
I20260808 18:31:57.890659 15719 mvcc.cc:205] Tried to move back new op lower bound from 7316332207626883072 to 7316332207520456704. Current Snapshot: MvccSnapshot[applied={T|T < 7316332207626883072}]
I20260808 18:31:57.899322 15189 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:31:57.899518 15189 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:31:57.902120 15189 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea: Sending DeleteTablet for 3 replicas of tablet 14728559464c4d8ebdd0f83df08a8924
I20260808 18:31:57.902940 15622 tablet_service.cc:1514] Processing DeleteTablet for tablet 14728559464c4d8ebdd0f83df08a8924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:31:57 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:50976
I20260808 18:31:57.902993 15475 tablet_service.cc:1514] Processing DeleteTablet for tablet 14728559464c4d8ebdd0f83df08a8924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:31:57 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:54058
I20260808 18:31:57.903111 15322 tablet_service.cc:1514] Processing DeleteTablet for tablet 14728559464c4d8ebdd0f83df08a8924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:31:57 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:35614
I20260808 18:31:57.903203 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 15232
I20260808 18:31:57.908844 15728 tablet_replica.cc:330] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e: stopping tablet replica
I20260808 18:31:57.909145 15728 raft_consensus.cc:2229] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:31:57.909507 15728 raft_consensus.cc:2258] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:31:57.910001 15729 tablet_replica.cc:330] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323: stopping tablet replica
I20260808 18:31:57.910293 15729 raft_consensus.cc:2229] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:31:57.910611 15729 raft_consensus.cc:2258] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:31:57.911175 15728 ts_tablet_manager.cc:1905] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:31:57.912014 15729 ts_tablet_manager.cc:1905] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 18:31:57.913443 15174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:44659 (error 108)
I20260808 18:31:57.914198 15728 ts_tablet_manager.cc:1918] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:31:57.914378 15728 log.cc:1196] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/14728559464c4d8ebdd0f83df08a8924
I20260808 18:31:57.914590 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 15400
I20260808 18:31:57.914896 15728 ts_tablet_manager.cc:1939] T 14728559464c4d8ebdd0f83df08a8924 P 960ac1fba0aa4862ada0a3652663025e: Deleting consensus metadata
I20260808 18:31:57.916198 15176 catalog_manager.cc:4841] TS 960ac1fba0aa4862ada0a3652663025e (127.29.78.131:45375): tablet 14728559464c4d8ebdd0f83df08a8924 (table test-table [id=eb07d04285944b9b856431fb833be5da]) successfully deleted
I20260808 18:31:57.923810 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:57.923382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:44659 (local address 127.0.0.1:35616)
0808 18:31:57.923644 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:57.923724 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:44659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
W20260808 18:31:57.923998 15174 catalog_manager.cc:4584] TS 78d85fdfa3db4e9abda1a406fbe28b73 (127.29.78.129:44659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 14728559464c4d8ebdd0f83df08a8924: Network error: Client connection negotiation failed: client connection to 127.29.78.129:44659: connect: Connection refused (error 111)
I20260808 18:31:57.928607 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 15551
I20260808 18:31:57.929123 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:31:57.928737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44727 (local address 127.0.0.1:54074)
0808 18:31:57.928955 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0808 18:31:57.929022 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.130:44727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260808 18:31:57.936060 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 15155
I20260808 18:32:17.944036 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:44257
--webserver_interface=127.29.78.190
--webserver_port=36637
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:44257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:18.056272 15732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:18.056577 15732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:18.056667 15732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:18.060804 15732 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 18:32:18.060880 15732 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 18:32:18.060907 15732 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 18:32:18.060932 15732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:32:18.061002 15732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:18.061045 15732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:18.061084 15732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:32:18.061120 15732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:32:18.066365 15732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:44257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:44257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=36637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:18.067552 15732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:18.068617 15732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:18.077809 15740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:18.077821 15737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:18.077863 15732 server_base.cc:1034] running on GCE node
W20260808 18:32:18.077798 15738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:18.078742 15732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:18.079880 15732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:18.081066 15732 hybrid_clock.cc:648] HybridClock initialized: now 1786213938081040 us; error 39 us; skew 500 ppm
Aug 08 18:32:18 dist-test-slave-gp6n krb5kdc[14551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213938, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:18.088514 15732 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:32:18.089632 15732 webserver.cc:460] Webserver started at http://127.29.78.190:36637/ using document root <none> and password file <none>
I20260808 18:32:18.090224 15732 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:18.090317 15732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:18.094133 15732 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:32:18.096359 15748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:18.097685 15732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 18:32:18.097841 15732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "047242e7bd634821bc96757a00bd1dea"
format_stamp: "Formatted at 2026-08-08 18:31:35 on dist-test-slave-gp6n"
server_key: "e3113693b33e4a523163d8e588301262"
server_key_iv: "33ff67cdf99bc2b4f26fc36f30049bc1"
server_key_version: "encryptionkey@0"
I20260808 18:32:18.098388 15732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:18.116091 15732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:18.119168 15732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:18.119376 15732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:18.126968 15732 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:44257
I20260808 18:32:18.127264 15810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:44257 every 8 connection(s)
I20260808 18:32:18.128072 15732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:32:18.131807 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 15732
I20260808 18:32:18.132256 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:44659
--local_ip_for_outbound_sockets=127.29.78.129
--tserver_master_addrs=127.29.78.190:44257
--webserver_port=37093
--webserver_interface=127.29.78.129
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:32:18.132560 15811 sys_catalog.cc:263] Verifying existing consensus state
I20260808 18:32:18.134194 15811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea: Bootstrap starting.
I20260808 18:32:18.151736 15811 log.cc:824] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:18.162765 15811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:32:18.163235 15811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea: Bootstrap complete.
I20260808 18:32:18.167303 15811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047242e7bd634821bc96757a00bd1dea" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 44257 } }
I20260808 18:32:18.167680 15811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 047242e7bd634821bc96757a00bd1dea, State: Initialized, Role: FOLLOWER
I20260808 18:32:18.168213 15811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047242e7bd634821bc96757a00bd1dea" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 44257 } }
I20260808 18:32:18.168331 15811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:32:18.168421 15811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:32:18.168540 15811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 2 FOLLOWER]: Advancing to term 3
I20260808 18:32:18.170464 15811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047242e7bd634821bc96757a00bd1dea" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 44257 } }
I20260808 18:32:18.170801 15811 leader_election.cc:302] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 047242e7bd634821bc96757a00bd1dea; no voters:
I20260808 18:32:18.171139 15811 leader_election.cc:288] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 18:32:18.171219 15815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 3 FOLLOWER]: Leader election won for term 3
I20260808 18:32:18.171465 15815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [term 3 LEADER]: Becoming Leader. State: Replica: 047242e7bd634821bc96757a00bd1dea, State: Running, Role: LEADER
I20260808 18:32:18.171806 15815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047242e7bd634821bc96757a00bd1dea" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 44257 } }
I20260808 18:32:18.172359 15811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:32:18.173223 15816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "047242e7bd634821bc96757a00bd1dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047242e7bd634821bc96757a00bd1dea" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 44257 } } }
I20260808 18:32:18.173344 15816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:18.173630 15817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 047242e7bd634821bc96757a00bd1dea. Latest consensus state: current_term: 3 leader_uuid: "047242e7bd634821bc96757a00bd1dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "047242e7bd634821bc96757a00bd1dea" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 44257 } } }
I20260808 18:32:18.173702 15817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:18.177284 15831 catalog_manager.cc:1259] Loaded cluster ID: b2d9f365084e4c51b109660263e5577a
I20260808 18:32:18.177345 15831 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea: loading cluster ID for follower catalog manager: success
I20260808 18:32:18.179188 15831 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 047242e7bd634821bc96757a00bd1dea: acquiring CA information for follower catalog manager: success
F20260808 18:32:18.180115 15831 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1cb6f789ad google::LogMessage::Fail() at ??:0
@ 0x7f1cb6f7b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f1cb6f784de google::LogMessage::Flush() at ??:0
@ 0x7f1cb6f79a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1cb9c3c18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1cc18dd989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f1cc18ba023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f1cc18ba54b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f1cc18b3652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f1cc18b29e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7f1cc18f9541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1cb91bb512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1cba7ff6db start_thread at ??:0
@ 0x7f1cb5a7661f clone at ??:0
W20260808 18:32:18.271207 15813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:18.271453 15813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:18.271495 15813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:18.275676 15813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:18.275733 15813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:18.275818 15813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:32:18.281046 15813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:44659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=37093
--tserver_master_addrs=127.29.78.190:44257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:18.282089 15813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:18.283142 15813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:18.292718 15854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:18.292697 15853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:18.292886 15813 server_base.cc:1034] running on GCE node
W20260808 18:32:18.293119 15856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:18.293530 15813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:18.294173 15813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:18.295347 15813 hybrid_clock.cc:648] HybridClock initialized: now 1786213938295311 us; error 51 us; skew 500 ppm
Aug 08 18:32:18 dist-test-slave-gp6n krb5kdc[14551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213938, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:18.298049 15813 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:32:18.299018 15813 webserver.cc:460] Webserver started at http://127.29.78.129:37093/ using document root <none> and password file <none>
I20260808 18:32:18.299556 15813 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:18.299619 15813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:18.303483 15813 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:32:18.305651 15864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:18.307020 15813 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260808 18:32:18.307142 15813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "78d85fdfa3db4e9abda1a406fbe28b73"
format_stamp: "Formatted at 2026-08-08 18:31:35 on dist-test-slave-gp6n"
server_key: "92e8ff621682108b6b24496ea7cbc2a1"
server_key_iv: "193b142c1770fc53af6b725ce3b52d44"
server_key_version: "encryptionkey@0"
I20260808 18:32:18.307667 15813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:18.323141 15813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:18.326279 15813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:18.326499 15813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:18.327096 15813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:32:18.328189 15871 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 18:32:18.333575 15813 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 18:32:18.333658 15813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260808 18:32:18.333739 15813 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 18:32:18.335419 15813 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 18:32:18.335491 15813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260808 18:32:18.335780 15871 tablet_bootstrap.cc:492] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73: Bootstrap starting.
I20260808 18:32:18.346176 15813 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:44659
I20260808 18:32:18.346315 15991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:44659 every 8 connection(s)
I20260808 18:32:18.347267 15813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:32:18.350139 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 15813
I20260808 18:32:18.350682 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:44727
--local_ip_for_outbound_sockets=127.29.78.130
--tserver_master_addrs=127.29.78.190:44257
--webserver_port=34837
--webserver_interface=127.29.78.130
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:32:18.357811 15871 log.cc:824] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:18.367244 15871 tablet_bootstrap.cc:492] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:32:18.367739 15871 tablet_bootstrap.cc:492] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73: Bootstrap complete.
I20260808 18:32:18.369207 15871 ts_tablet_manager.cc:1397] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73: Time spent bootstrapping tablet: real 0.034s user 0.027s sys 0.005s
I20260808 18:32:18.372587 15871 raft_consensus.cc:348] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:32:18.373260 15871 raft_consensus.cc:729] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d85fdfa3db4e9abda1a406fbe28b73, State: Initialized, Role: FOLLOWER
I20260808 18:32:18.373864 15871 consensus_queue.cc:260] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:32:18.375006 15871 ts_tablet_manager.cc:1428] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260808 18:32:18.392001 15994 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:32:18.349200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.129:46841)
0808 18:32:18.349699 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:18.349753 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:18.351592 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:18.391038 (+ 39446us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":174,"threads_started":1}
W20260808 18:32:18.392573 15992 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:44257 (0 consecutive failures): Network error: Failed to ping master at 127.29.78.190:44257: Client connection negotiation failed: client connection to 127.29.78.190:44257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 18:32:18.393432 15994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:18.393168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.129:60091)
0808 18:32:18.393308 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:18.393379 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 18:32:18.394158 15994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:18.393975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.129:48903)
0808 18:32:18.394072 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:18.394113 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 18:32:18.394467 15992 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 18:32:18.472167 15995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:18.472398 15995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:18.472440 15995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:18.476346 15995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:18.476398 15995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:18.476493 15995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:32:18.481499 15995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:44727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=34837
--tserver_master_addrs=127.29.78.190:44257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:18.482458 15995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:18.483431 15995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:18.493000 16004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:18.492997 16005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:18.493263 15995 server_base.cc:1034] running on GCE node
W20260808 18:32:18.493427 16007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:18.493861 15995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:18.494457 15995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:18.495636 15995 hybrid_clock.cc:648] HybridClock initialized: now 1786213938495619 us; error 27 us; skew 500 ppm
Aug 08 18:32:18 dist-test-slave-gp6n krb5kdc[14551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213938, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:18.498373 15995 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:32:18.499240 15995 webserver.cc:460] Webserver started at http://127.29.78.130:34837/ using document root <none> and password file <none>
I20260808 18:32:18.499786 15995 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:18.499878 15995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:18.503553 15995 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:32:18.505574 16014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:18.506815 15995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:32:18.506943 15995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "08c941ac7e764e27abd46653e2b8c323"
format_stamp: "Formatted at 2026-08-08 18:31:36 on dist-test-slave-gp6n"
server_key: "ee1d5edf3ef279a2eb30fcd7aab35409"
server_key_iv: "3a0ff046b1decd1f1859f7d66c5673b6"
server_key_version: "encryptionkey@0"
I20260808 18:32:18.507488 15995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:18.533213 15995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:18.537130 15995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:18.537344 15995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:18.538143 15995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:32:18.539279 16021 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260808 18:32:18.542985 16021 ts_tablet_manager.cc:1852] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260808 18:32:18.543372 16021 ts_tablet_manager.cc:1905] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:32:18.547317 16021 ts_tablet_manager.cc:1918] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:32:18.547410 16021 log.cc:1196] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/14728559464c4d8ebdd0f83df08a8924
I20260808 18:32:18.547840 16021 ts_tablet_manager.cc:1939] T 14728559464c4d8ebdd0f83df08a8924 P 08c941ac7e764e27abd46653e2b8c323: Deleting consensus metadata
I20260808 18:32:18.548164 15995 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260808 18:32:18.548242 15995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260808 18:32:18.548345 15995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:32:18.548388 15995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:18.557353 15995 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:44727
I20260808 18:32:18.557397 16143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:44727 every 8 connection(s)
I20260808 18:32:18.558214 15995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:32:18.559165 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 15995
I20260808 18:32:18.559593 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:45375
--local_ip_for_outbound_sockets=127.29.78.131
--tserver_master_addrs=127.29.78.190:44257
--webserver_port=33911
--webserver_interface=127.29.78.131
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:32:18.562364 16147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:18.560417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.130:46855)
0808 18:32:18.561584 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:18.561690 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":959,"thread_start_us":112,"threads_started":1}
W20260808 18:32:18.562839 16145 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:44257 (0 consecutive failures): Network error: Failed to ping master at 127.29.78.190:44257: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
I20260808 18:32:18.563485 16147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:18.563183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.130:59649)
0808 18:32:18.563336 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:18.563402 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 18:32:18.564227 16147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:18.563920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.130:56247)
0808 18:32:18.564079 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:18.564147 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 18:32:18.564406 16145 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 18:32:18.671342 16149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:18.671593 16149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:18.671636 16149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:18.675452 16149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:18.675503 16149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:18.675563 16149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:32:18.680511 16149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:45375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=33911
--tserver_master_addrs=127.29.78.190:44257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:18.681429 16149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:18.682363 16149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:18.691665 16157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:18.691648 16159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:18.691625 16156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:18.691988 16149 server_base.cc:1034] running on GCE node
I20260808 18:32:18.692373 16149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:18.693002 16149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:18.694193 16149 hybrid_clock.cc:648] HybridClock initialized: now 1786213938694174 us; error 30 us; skew 500 ppm
Aug 08 18:32:18 dist-test-slave-gp6n krb5kdc[14551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213938, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:18.696866 16149 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:32:18.697693 16149 webserver.cc:460] Webserver started at http://127.29.78.131:33911/ using document root <none> and password file <none>
I20260808 18:32:18.698244 16149 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:18.698336 16149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:18.702004 16149 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:32:18.703996 16168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:18.705137 16149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:32:18.705269 16149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "960ac1fba0aa4862ada0a3652663025e"
format_stamp: "Formatted at 2026-08-08 18:31:36 on dist-test-slave-gp6n"
server_key: "bb83dc82a2cd7050f82ebebc3a124aea"
server_key_iv: "e40ff0f884b62658c7f51d3c80d91481"
server_key_version: "encryptionkey@0"
I20260808 18:32:18.705806 16149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:18.721748 16149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:18.724550 16149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:18.724743 16149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:18.725324 16149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:32:18.726123 16149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:32:18.726190 16149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:18.726253 16149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:32:18.726293 16149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:18.734802 16149 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:45375
I20260808 18:32:18.734825 16295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:45375 every 8 connection(s)
I20260808 18:32:18.735774 16149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:32:18.736941 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 16149
I20260808 18:32:18.739933 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:18.738081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.131:42963)
0808 18:32:18.738624 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:18.738724 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":128,"threads_started":1}
W20260808 18:32:18.740435 16296 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:44257 (0 consecutive failures): Network error: Failed to ping master at 127.29.78.190:44257: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
I20260808 18:32:18.740989 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:18.740791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.131:36605)
0808 18:32:18.740892 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:18.740937 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260808 18:32:18.741562 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:18.741378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.131:59531)
0808 18:32:18.741463 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:18.741502 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260808 18:32:18.741720 16296 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 18:32:19.395517 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:19.395088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.129:57867)
0808 18:32:19.395347 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:19.395405 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
I20260808 18:32:19.565333 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:19.564973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.130:45877)
0808 18:32:19.565200 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:19.565252 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":70,"threads_started":1}
I20260808 18:32:19.742738 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:19.742250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.131:55089)
0808 18:32:19.742539 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:19.742607 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":71,"threads_started":1}
I20260808 18:32:20.155478 16304 raft_consensus.cc:482] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:32:20.155723 16304 raft_consensus.cc:504] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:32:20.156842 16304 leader_election.cc:288] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 960ac1fba0aa4862ada0a3652663025e (127.29.78.131:45375), 08c941ac7e764e27abd46653e2b8c323 (127.29.78.130:44727)
Aug 08 18:32:20 dist-test-slave-gp6n krb5kdc[14551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213938, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.131@KRBTEST.COM
Aug 08 18:32:20 dist-test-slave-gp6n krb5kdc[14551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213938, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.130@KRBTEST.COM
I20260808 18:32:20.173717 16307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:20.157559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44727 (local address 127.29.78.129:54443)
0808 18:32:20.157928 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:20.157944 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:20.158043 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:20.159677 (+ 1634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:20.159683 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:20.160335 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:32:20.161230 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:20.161253 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:20.163278 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:20.163281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:20.164553 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:20.164565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:20.165539 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:20.165543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:20.165856 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:20.166747 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:32:20.166775 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:32:20.169032 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:32:20.171291 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:32:20.171294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:32:20.171295 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:32:20.171609 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:32:20.171984 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:32:20.171986 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:32:20.171987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:32:20.172094 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:32:20.172714 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:32:20.172720 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:32:20.172988 (+ 268us) client_negotiation.cc:769] Sending connection context
0808 18:32:20.173170 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 18:32:20.173525 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":296,"spinlock_wait_cycles":36864,"thread_start_us":88,"threads_started":1}
I20260808 18:32:20.173728 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:20.157278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:57063 (local address 127.29.78.131:45375)
0808 18:32:20.157730 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:20.157751 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:20.157792 (+ 41us) server_negotiation.cc:407] Connection header received
0808 18:32:20.158735 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:20.158745 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:20.159208 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:20.159570 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:20.161436 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:20.163058 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:20.164684 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:20.165394 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:20.165569 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:20.169052 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:20.169061 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:20.169081 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:20.169120 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:20.171027 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:20.171701 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:20.171708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:20.171715 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:20.171837 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:20.172223 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:20.172226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:20.172227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:20.172590 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:20.172724 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:20.173285 (+ 561us) server_negotiation.cc:299] Negotiation successful
0808 18:32:20.173539 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":104,"threads_started":1}
I20260808 18:32:20.174007 16308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:20.157754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:54443 (local address 127.29.78.130:44727)
0808 18:32:20.158120 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:20.158133 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:20.158168 (+ 35us) server_negotiation.cc:407] Connection header received
0808 18:32:20.159036 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:20.159045 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:20.159375 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:20.159627 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:20.161436 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:20.163058 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:20.164685 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:20.165394 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:20.165569 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:20.169205 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:20.169210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:20.169220 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:20.169245 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:20.171178 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:20.171736 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:20.171740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:20.171744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:20.171851 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:20.172189 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:20.172194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:20.172196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:20.172586 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:20.172719 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:20.173293 (+ 574us) server_negotiation.cc:299] Negotiation successful
0808 18:32:20.173560 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":147,"threads_started":1}
I20260808 18:32:20.173767 16305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:20.157114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:45375 (local address 127.29.78.129:57063)
0808 18:32:20.157381 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:20.157398 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:20.157515 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:20.159677 (+ 2162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:20.159683 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:20.160340 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:32:20.161230 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:20.161253 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:20.163702 (+ 2449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:20.163705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:20.164553 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:20.164565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:20.165539 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:20.165543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:20.165857 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:20.166747 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:32:20.166775 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:32:20.168822 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:32:20.171149 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:32:20.171156 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:32:20.171169 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:32:20.171589 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:32:20.171947 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:32:20.171951 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:32:20.171952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:32:20.172107 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:32:20.172716 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:32:20.172722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:32:20.172987 (+ 265us) client_negotiation.cc:769] Sending connection context
0808 18:32:20.173171 (+ 184us) client_negotiation.cc:240] Negotiation successful
0808 18:32:20.173611 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":457,"spinlock_wait_cycles":40448,"thread_start_us":87,"threads_started":1}
W20260808 18:32:20.177103 15868 leader_election.cc:341] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 08c941ac7e764e27abd46653e2b8c323 (127.29.78.130:44727): Not found: Tablet not found: 14728559464c4d8ebdd0f83df08a8924
W20260808 18:32:20.177321 15868 leader_election.cc:341] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 960ac1fba0aa4862ada0a3652663025e (127.29.78.131:45375): Not found: Tablet not found: 14728559464c4d8ebdd0f83df08a8924
I20260808 18:32:20.177378 15868 leader_election.cc:302] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78d85fdfa3db4e9abda1a406fbe28b73; no voters: 08c941ac7e764e27abd46653e2b8c323, 960ac1fba0aa4862ada0a3652663025e
I20260808 18:32:20.177666 16304 raft_consensus.cc:2726] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 18:32:20.396667 16305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:20.396312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.129:60161)
0808 18:32:20.396491 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:20.396555 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260808 18:32:20.566622 16309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:20.566100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.130:38749)
0808 18:32:20.566460 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:20.566522 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":101,"threads_started":1}
I20260808 18:32:20.743824 16310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:20.743448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.131:49685)
0808 18:32:20.743687 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:20.743739 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
I20260808 18:32:21.397883 16311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:21.397453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.129:40329)
0808 18:32:21.397723 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:21.397786 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
I20260808 18:32:21.567749 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:21.567312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.130:47985)
0808 18:32:21.567587 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:21.567643 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":86,"threads_started":1}
I20260808 18:32:21.745122 16313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:21.744627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.131:40679)
0808 18:32:21.744904 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:21.744962 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":102,"threads_started":1}
I20260808 18:32:22.399272 16314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:22.398847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.129:49555)
0808 18:32:22.399116 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:22.399176 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":78,"threads_started":1}
I20260808 18:32:22.421427 16316 raft_consensus.cc:482] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:32:22.421558 16316 raft_consensus.cc:504] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:32:22.421928 16316 leader_election.cc:288] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 960ac1fba0aa4862ada0a3652663025e (127.29.78.131:45375), 08c941ac7e764e27abd46653e2b8c323 (127.29.78.130:44727)
W20260808 18:32:22.422827 15868 leader_election.cc:341] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 960ac1fba0aa4862ada0a3652663025e (127.29.78.131:45375): Not found: Tablet not found: 14728559464c4d8ebdd0f83df08a8924
W20260808 18:32:22.423010 15868 leader_election.cc:341] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 08c941ac7e764e27abd46653e2b8c323 (127.29.78.130:44727): Not found: Tablet not found: 14728559464c4d8ebdd0f83df08a8924
I20260808 18:32:22.423074 15868 leader_election.cc:302] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78d85fdfa3db4e9abda1a406fbe28b73; no voters: 08c941ac7e764e27abd46653e2b8c323, 960ac1fba0aa4862ada0a3652663025e
I20260808 18:32:22.423254 16316 raft_consensus.cc:2726] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 18:32:22.568926 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:22.568522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.130:55589)
0808 18:32:22.568770 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:22.568823 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":91,"threads_started":1}
I20260808 18:32:22.746270 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:22.745855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.131:36543)
0808 18:32:22.746121 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:22.746178 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":92,"threads_started":1}
I20260808 18:32:23.400856 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:23.400273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.129:38069)
0808 18:32:23.400676 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:23.400738 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":162,"threads_started":1}
I20260808 18:32:23.570150 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:23.569734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.130:41549)
0808 18:32:23.570010 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:23.570065 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
I20260808 18:32:23.747691 16321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:23.747095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.131:53663)
0808 18:32:23.747544 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:23.747600 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":251,"threads_started":1}
I20260808 18:32:24.402189 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:24.401721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.129:56057)
0808 18:32:24.402030 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:24.402092 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":98,"threads_started":1}
I20260808 18:32:24.571468 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:24.571017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.130:53507)
0808 18:32:24.571322 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:24.571378 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":115,"threads_started":1}
I20260808 18:32:24.748944 16325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:24.748468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.131:35133)
0808 18:32:24.748783 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:24.748844 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":94,"threads_started":1}
I20260808 18:32:25.403647 16326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:25.403155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.129:47517)
0808 18:32:25.403473 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:25.403537 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":100,"threads_started":1}
I20260808 18:32:25.472711 16327 raft_consensus.cc:482] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:32:25.472879 16327 raft_consensus.cc:504] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:32:25.473312 16327 leader_election.cc:288] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 960ac1fba0aa4862ada0a3652663025e (127.29.78.131:45375), 08c941ac7e764e27abd46653e2b8c323 (127.29.78.130:44727)
W20260808 18:32:25.474151 15868 leader_election.cc:341] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 960ac1fba0aa4862ada0a3652663025e (127.29.78.131:45375): Not found: Tablet not found: 14728559464c4d8ebdd0f83df08a8924
W20260808 18:32:25.474304 15868 leader_election.cc:341] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 08c941ac7e764e27abd46653e2b8c323 (127.29.78.130:44727): Not found: Tablet not found: 14728559464c4d8ebdd0f83df08a8924
I20260808 18:32:25.474360 15868 leader_election.cc:302] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78d85fdfa3db4e9abda1a406fbe28b73; no voters: 08c941ac7e764e27abd46653e2b8c323, 960ac1fba0aa4862ada0a3652663025e
I20260808 18:32:25.474545 16327 raft_consensus.cc:2726] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 18:32:25.572644 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:25.572165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.130:46259)
0808 18:32:25.572499 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:25.572555 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":71,"threads_started":1}
I20260808 18:32:25.750172 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:25.749704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.131:57527)
0808 18:32:25.750028 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:25.750085 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":72,"threads_started":1}
I20260808 18:32:26.405009 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:26.404558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.129:39413)
0808 18:32:26.404848 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:26.404909 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":104,"threads_started":1}
I20260808 18:32:26.573772 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:26.573399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.130:37681)
0808 18:32:26.573632 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:26.573685 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
I20260808 18:32:26.751346 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:26.750949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.131:60791)
0808 18:32:26.751197 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:26.751253 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
I20260808 18:32:27.406241 16333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:27.405807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.129:51773)
0808 18:32:27.406087 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:27.406142 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":99,"threads_started":1}
I20260808 18:32:27.575045 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:27.574610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.130:34321)
0808 18:32:27.574884 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:27.574947 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
I20260808 18:32:27.752501 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:27.752088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.131:50385)
0808 18:32:27.752343 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:27.752406 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
I20260808 18:32:28.407637 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:28.407217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.129:54335)
0808 18:32:28.407481 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:28.407540 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
I20260808 18:32:28.438953 16338 raft_consensus.cc:482] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:32:28.439114 16338 raft_consensus.cc:504] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:32:28.439574 16338 leader_election.cc:288] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 960ac1fba0aa4862ada0a3652663025e (127.29.78.131:45375), 08c941ac7e764e27abd46653e2b8c323 (127.29.78.130:44727)
W20260808 18:32:28.440354 15868 leader_election.cc:341] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 960ac1fba0aa4862ada0a3652663025e (127.29.78.131:45375): Not found: Tablet not found: 14728559464c4d8ebdd0f83df08a8924
W20260808 18:32:28.440496 15868 leader_election.cc:341] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 08c941ac7e764e27abd46653e2b8c323 (127.29.78.130:44727): Not found: Tablet not found: 14728559464c4d8ebdd0f83df08a8924
I20260808 18:32:28.440562 15868 leader_election.cc:302] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78d85fdfa3db4e9abda1a406fbe28b73; no voters: 08c941ac7e764e27abd46653e2b8c323, 960ac1fba0aa4862ada0a3652663025e
I20260808 18:32:28.440743 16338 raft_consensus.cc:2726] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 18:32:28.576202 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:28.575808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.130:60127)
0808 18:32:28.576056 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:28.576107 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
I20260808 18:32:28.753648 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:28.753248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.131:42069)
0808 18:32:28.753502 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:28.753556 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
I20260808 18:32:29.408814 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:29.408362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.129:53289)
0808 18:32:29.408659 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:29.408716 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":82,"threads_started":1}
I20260808 18:32:29.577466 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:29.577009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.130:40471)
0808 18:32:29.577324 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:29.577373 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":96,"threads_started":1}
I20260808 18:32:29.754820 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:29.754367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.131:58311)
0808 18:32:29.754675 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:29.754729 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":136,"threads_started":1}
I20260808 18:32:30.410179 16344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:30.409699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.129:44247)
0808 18:32:30.410019 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:30.410083 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":79,"threads_started":1}
I20260808 18:32:30.578681 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:30.578203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.130:57387)
0808 18:32:30.578531 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:30.578588 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":83,"threads_started":1}
I20260808 18:32:30.756093 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:30.755669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.131:33919)
0808 18:32:30.755929 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:30.755988 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":84,"threads_started":1}
I20260808 18:32:31.411578 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:31.411111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.129:47709)
0808 18:32:31.411379 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:31.411442 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":77,"threads_started":1}
I20260808 18:32:31.579787 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:31.579387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.130:57131)
0808 18:32:31.579641 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:31.579694 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
I20260808 18:32:31.757369 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:31.756926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.131:53297)
0808 18:32:31.757217 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:31.757274 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":79,"threads_started":1}
I20260808 18:32:32.412729 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:32.412320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.129:45819)
0808 18:32:32.412577 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:32.412633 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":76,"threads_started":1}
I20260808 18:32:32.580992 16352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:32.580557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.130:38343)
0808 18:32:32.580833 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:32.580895 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":79,"threads_started":1}
I20260808 18:32:32.758517 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:32.758049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.131:52417)
0808 18:32:32.758313 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:32.758364 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":92,"threads_started":1}
I20260808 18:32:32.838119 16354 raft_consensus.cc:482] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:32:32.838258 16354 raft_consensus.cc:504] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "960ac1fba0aa4862ada0a3652663025e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45375 } } peers { permanent_uuid: "08c941ac7e764e27abd46653e2b8c323" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44727 } } peers { permanent_uuid: "78d85fdfa3db4e9abda1a406fbe28b73" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44659 } }
I20260808 18:32:32.838733 16354 leader_election.cc:288] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 960ac1fba0aa4862ada0a3652663025e (127.29.78.131:45375), 08c941ac7e764e27abd46653e2b8c323 (127.29.78.130:44727)
W20260808 18:32:32.839639 15868 leader_election.cc:341] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 08c941ac7e764e27abd46653e2b8c323 (127.29.78.130:44727): Not found: Tablet not found: 14728559464c4d8ebdd0f83df08a8924
W20260808 18:32:32.839776 15868 leader_election.cc:341] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 960ac1fba0aa4862ada0a3652663025e (127.29.78.131:45375): Not found: Tablet not found: 14728559464c4d8ebdd0f83df08a8924
I20260808 18:32:32.839841 15868 leader_election.cc:302] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78d85fdfa3db4e9abda1a406fbe28b73; no voters: 08c941ac7e764e27abd46653e2b8c323, 960ac1fba0aa4862ada0a3652663025e
I20260808 18:32:32.840005 16354 raft_consensus.cc:2726] T 14728559464c4d8ebdd0f83df08a8924 P 78d85fdfa3db4e9abda1a406fbe28b73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 18:32:33.414150 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:33.413690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.129:56669)
0808 18:32:33.413999 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:33.414056 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":131,"threads_started":1}
I20260808 18:32:33.582309 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:33.581882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:44257 (local address 127.29.78.130:56843)
0808 18:32:33.582159 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:33.582214 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":83,"threads_started":1}
I20260808 18:32:33.738171 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 15813
I20260808 18:32:33.747555 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 15995
I20260808 18:32:33.755791 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 16149
2026-08-08T18:32:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59521 ms)
[----------] 30 tests from SecurityITest (244974 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:32:33 dist-test-slave-gp6n krb5kdc[16360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:32:33 dist-test-slave-gp6n krb5kdc[16360](info): set up 2 sockets
Aug 08 18:32:33 dist-test-slave-gp6n krb5kdc[16360](info): commencing operation
krb5kdc: starting...
W20260808 18:32:34.899456 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:32:34 dist-test-slave-gp6n krb5kdc[16360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:32:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:32:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:32:35.025785 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:41371
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:41371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:35.142076 16374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:35.142328 16374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:35.142367 16374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:35.146348 16374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:32:35.146421 16374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:35.146442 16374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:35.146458 16374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:32:35.146509 16374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:32:35.152961 16374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:41371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:41371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:35.154018 16374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:35.155107 16374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:35.164165 16383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:35.164209 16380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:35.164182 16374 server_base.cc:1034] running on GCE node
W20260808 18:32:35.164204 16381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:35.164997 16374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:35.166024 16374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:35.167217 16374 hybrid_clock.cc:648] HybridClock initialized: now 1786213955167205 us; error 27 us; skew 500 ppm
Aug 08 18:32:35 dist-test-slave-gp6n krb5kdc[16360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:35.170020 16374 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:32:35.170902 16374 webserver.cc:460] Webserver started at http://127.29.78.190:43327/ using document root <none> and password file <none>
I20260808 18:32:35.171415 16374 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:35.171473 16374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:35.171634 16374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:35.173689 16374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "a907c74ebe024aec85f636f2de05f69b"
format_stamp: "Formatted at 2026-08-08 18:32:35 on dist-test-slave-gp6n"
server_key: "942c14d6c1dc3ad73c7018ae49a111cb"
server_key_iv: "b7ccf774e9eefc3a65ce262c54a5ebd9"
server_key_version: "encryptionkey@0"
I20260808 18:32:35.174201 16374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "a907c74ebe024aec85f636f2de05f69b"
format_stamp: "Formatted at 2026-08-08 18:32:35 on dist-test-slave-gp6n"
server_key: "942c14d6c1dc3ad73c7018ae49a111cb"
server_key_iv: "b7ccf774e9eefc3a65ce262c54a5ebd9"
server_key_version: "encryptionkey@0"
I20260808 18:32:35.177800 16374 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:32:35.180178 16392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:35.181368 16374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 18:32:35.181627 16374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "a907c74ebe024aec85f636f2de05f69b"
format_stamp: "Formatted at 2026-08-08 18:32:35 on dist-test-slave-gp6n"
server_key: "942c14d6c1dc3ad73c7018ae49a111cb"
server_key_iv: "b7ccf774e9eefc3a65ce262c54a5ebd9"
server_key_version: "encryptionkey@0"
I20260808 18:32:35.181728 16374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:35.203351 16374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:35.206214 16374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:35.206449 16374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:35.213584 16374 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:41371
I20260808 18:32:35.213615 16446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:41371 every 8 connection(s)
I20260808 18:32:35.214576 16374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:32:35.217315 16447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:35.222935 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 16374
I20260808 18:32:35.223073 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:32:35.223409 30010 external_mini_cluster.cc:1428] Setting key be063efcebf610fd165a3284638b3be1
I20260808 18:32:35.223469 16447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b: Bootstrap starting.
I20260808 18:32:35.226130 16447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:35.226964 16447 log.cc:824] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:35.229164 16447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b: No bootstrap required, opened a new log
Aug 08 18:32:35 dist-test-slave-gp6n krb5kdc[16360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:32:35.233883 16447 raft_consensus.cc:348] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a907c74ebe024aec85f636f2de05f69b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41371 } }
I20260808 18:32:35.234154 16447 raft_consensus.cc:374] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:35.234233 16447 raft_consensus.cc:729] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a907c74ebe024aec85f636f2de05f69b, State: Initialized, Role: FOLLOWER
I20260808 18:32:35.234776 16447 consensus_queue.cc:260] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a907c74ebe024aec85f636f2de05f69b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41371 } }
I20260808 18:32:35.234934 16447 raft_consensus.cc:388] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:32:35.235028 16447 raft_consensus.cc:482] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:32:35.235131 16447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:35.236378 16447 raft_consensus.cc:504] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a907c74ebe024aec85f636f2de05f69b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41371 } }
I20260808 18:32:35.236752 16447 leader_election.cc:302] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a907c74ebe024aec85f636f2de05f69b; no voters:
I20260808 18:32:35.237177 16447 leader_election.cc:288] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:32:35.237295 16452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:32:35.237535 16452 raft_consensus.cc:686] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b [term 1 LEADER]: Becoming Leader. State: Replica: a907c74ebe024aec85f636f2de05f69b, State: Running, Role: LEADER
I20260808 18:32:35.237879 16452 consensus_queue.cc:237] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a907c74ebe024aec85f636f2de05f69b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41371 } }
I20260808 18:32:35.238670 16447 sys_catalog.cc:564] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:32:35.239792 16454 sys_catalog.cc:455] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a907c74ebe024aec85f636f2de05f69b. Latest consensus state: current_term: 1 leader_uuid: "a907c74ebe024aec85f636f2de05f69b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a907c74ebe024aec85f636f2de05f69b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41371 } } }
I20260808 18:32:35.239795 16453 sys_catalog.cc:455] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a907c74ebe024aec85f636f2de05f69b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a907c74ebe024aec85f636f2de05f69b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41371 } } }
I20260808 18:32:35.239935 16450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:35.224951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46942 (local address 127.29.78.190:41371)
0808 18:32:35.225449 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:35.225467 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:35.225507 (+ 40us) server_negotiation.cc:407] Connection header received
0808 18:32:35.226313 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:35.226334 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:35.226815 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:35.227223 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:35.228077 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.229131 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.230337 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.230919 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.231075 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:35.234079 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:35.234085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:35.234099 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:35.234127 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:35.236383 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:35.236892 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:35.236898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:35.236904 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:35.237068 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:35.237441 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:35.237448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:35.237453 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:35.238021 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:35.238215 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:35.238551 (+ 336us) server_negotiation.cc:299] Negotiation successful
0808 18:32:35.238876 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":146,"threads_started":1}
I20260808 18:32:35.239964 16453 sys_catalog.cc:458] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:35.239964 16454 sys_catalog.cc:458] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b [sys.catalog]: This master's current role is: LEADER
W20260808 18:32:35.243814 16469 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:32:35.243952 16469 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:32:35.244035 16463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:32:35.244896 16463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:32:35.250554 16463 catalog_manager.cc:1347] Generated new cluster ID: 788e6f8dbdd242fe8981076fba8c1bf7
I20260808 18:32:35.250620 16463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:32:35.274919 16463 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:32:35.275926 16463 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:32:35.280233 16463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b: Generated new TSK 0
I20260808 18:32:35.281051 16463 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260808 18:32:35.317497 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41371
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:35.430231 16477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:35.430508 16477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:35.430552 16477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:35.434450 16477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:35.434499 16477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:35.434554 16477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:32:35.439829 16477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:35.440893 16477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:35.441900 16477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:32:35.451280 16477 server_base.cc:1034] running on GCE node
W20260808 18:32:35.451246 16485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:35.451242 16482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:35.451242 16483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:35.452064 16477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:35.452764 16477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:35.453925 16477 hybrid_clock.cc:648] HybridClock initialized: now 1786213955453907 us; error 34 us; skew 500 ppm
Aug 08 18:32:35 dist-test-slave-gp6n krb5kdc[16360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:35.456912 16477 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:32:35.457864 16477 webserver.cc:460] Webserver started at http://127.29.78.129:46203/ using document root <none> and password file <none>
I20260808 18:32:35.458508 16477 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:35.458578 16477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:35.458797 16477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:35.460850 16477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "bd99100fd60e4acbb2a98a310bf11514"
format_stamp: "Formatted at 2026-08-08 18:32:35 on dist-test-slave-gp6n"
server_key: "fdb174a03767654312187bec836cabd4"
server_key_iv: "f5a05d3483ec419df1af3bd0f3543ae1"
server_key_version: "encryptionkey@0"
I20260808 18:32:35.461428 16477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "bd99100fd60e4acbb2a98a310bf11514"
format_stamp: "Formatted at 2026-08-08 18:32:35 on dist-test-slave-gp6n"
server_key: "fdb174a03767654312187bec836cabd4"
server_key_iv: "f5a05d3483ec419df1af3bd0f3543ae1"
server_key_version: "encryptionkey@0"
I20260808 18:32:35.465250 16477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 18:32:35.467651 16493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:35.468852 16477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:32:35.468974 16477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "bd99100fd60e4acbb2a98a310bf11514"
format_stamp: "Formatted at 2026-08-08 18:32:35 on dist-test-slave-gp6n"
server_key: "fdb174a03767654312187bec836cabd4"
server_key_iv: "f5a05d3483ec419df1af3bd0f3543ae1"
server_key_version: "encryptionkey@0"
I20260808 18:32:35.469087 16477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:35.478350 16477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:35.481580 16477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:35.481727 16477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:35.482344 16477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:32:35.483245 16477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:32:35.483292 16477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:35.483361 16477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:32:35.483392 16477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:35.492339 16477 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:42675
I20260808 18:32:35.492408 16614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:42675 every 8 connection(s)
I20260808 18:32:35.493238 16477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:32:35.494055 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 16477
I20260808 18:32:35.494201 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:32:35.494592 30010 external_mini_cluster.cc:1428] Setting key d79b5e8a1d4d4f69383251c6a94681fe
Aug 08 18:32:35 dist-test-slave-gp6n krb5kdc[16360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
I20260808 18:32:35.507851 16450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:35.495225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:57631 (local address 127.29.78.190:41371)
0808 18:32:35.495375 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:35.495394 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:35.496869 (+ 1475us) server_negotiation.cc:407] Connection header received
0808 18:32:35.497776 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:35.497779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:35.497831 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:35.497939 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:35.499205 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.499739 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.501002 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.501241 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.501354 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:35.503914 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:35.503917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:35.503919 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:35.503941 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:35.505455 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:35.506124 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:35.506129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:35.506131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:35.506216 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:35.506631 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:35.506637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:35.506642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:35.506881 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:35.506992 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:35.507524 (+ 532us) server_negotiation.cc:299] Negotiation successful
0808 18:32:35.507667 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 18:32:35.508635 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:35.495667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:41371 (local address 127.29.78.129:57631)
0808 18:32:35.496716 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:35.496761 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:35.497574 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:35.498121 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:35.498128 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:35.498568 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:32:35.499042 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:35.499063 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.499871 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.499875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:35.500872 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:35.500882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.501397 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.501401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:35.501571 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:35.502215 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:32:35.502231 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:32:35.503742 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:32:35.505594 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:32:35.505601 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:32:35.505619 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:32:35.506009 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:32:35.506315 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:32:35.506318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:32:35.506321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:32:35.506530 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:32:35.507001 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:32:35.507010 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:32:35.507282 (+ 272us) client_negotiation.cc:769] Sending connection context
0808 18:32:35.507464 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 18:32:35.507790 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"thread_start_us":733,"threads_started":1}
I20260808 18:32:35.509915 16615 heartbeater.cc:347] Connected to a master server at 127.29.78.190:41371
I20260808 18:32:35.510174 16615 heartbeater.cc:464] Registering TS with master...
I20260808 18:32:35.510715 16615 heartbeater.cc:511] Master 127.29.78.190:41371 requested a full tablet report, sending...
I20260808 18:32:35.512434 16410 ts_manager.cc:194] Registered new tserver with Master: bd99100fd60e4acbb2a98a310bf11514 (127.29.78.129:42675)
I20260808 18:32:35.514178 16410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:57631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260808 18:32:35.525331 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41371
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:35.637521 16621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:35.637761 16621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:35.637804 16621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:35.641979 16621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:35.642040 16621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:35.642097 16621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:32:35.647468 16621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:35.648515 16621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:35.649547 16621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:35.659231 16626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:35.659215 16627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:35.659238 16629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:35.659499 16621 server_base.cc:1034] running on GCE node
I20260808 18:32:35.659843 16621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:35.660444 16621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:35.661602 16621 hybrid_clock.cc:648] HybridClock initialized: now 1786213955661590 us; error 25 us; skew 500 ppm
Aug 08 18:32:35 dist-test-slave-gp6n krb5kdc[16360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:35.664316 16621 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:32:35.665129 16621 webserver.cc:460] Webserver started at http://127.29.78.130:41987/ using document root <none> and password file <none>
I20260808 18:32:35.665669 16621 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:35.665731 16621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:35.665978 16621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:35.668021 16621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "b64dba75c59c48d987967d03d113d4af"
format_stamp: "Formatted at 2026-08-08 18:32:35 on dist-test-slave-gp6n"
server_key: "53fdfdbac18a46159ae313f8ce1495e1"
server_key_iv: "19253e054db655f7b866b2c33df39a64"
server_key_version: "encryptionkey@0"
I20260808 18:32:35.668592 16621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "b64dba75c59c48d987967d03d113d4af"
format_stamp: "Formatted at 2026-08-08 18:32:35 on dist-test-slave-gp6n"
server_key: "53fdfdbac18a46159ae313f8ce1495e1"
server_key_iv: "19253e054db655f7b866b2c33df39a64"
server_key_version: "encryptionkey@0"
I20260808 18:32:35.672536 16621 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:32:35.675236 16637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:35.676508 16621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:32:35.676643 16621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "b64dba75c59c48d987967d03d113d4af"
format_stamp: "Formatted at 2026-08-08 18:32:35 on dist-test-slave-gp6n"
server_key: "53fdfdbac18a46159ae313f8ce1495e1"
server_key_iv: "19253e054db655f7b866b2c33df39a64"
server_key_version: "encryptionkey@0"
I20260808 18:32:35.676759 16621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:35.697170 16621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:35.699868 16621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:35.700073 16621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:35.700634 16621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:32:35.701439 16621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:32:35.701508 16621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:35.701576 16621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:32:35.701612 16621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:35.710963 16621 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:44271
I20260808 18:32:35.710994 16758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:44271 every 8 connection(s)
I20260808 18:32:35.711817 16621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:32:35.712134 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 16621
I20260808 18:32:35.712271 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:32:35.712648 30010 external_mini_cluster.cc:1428] Setting key 79d7d790eba06c3fb0c939d2e43ebfcb
Aug 08 18:32:35 dist-test-slave-gp6n krb5kdc[16360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260808 18:32:35.726621 16450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:35.713780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:60335 (local address 127.29.78.190:41371)
0808 18:32:35.713921 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:35.713926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:35.715027 (+ 1101us) server_negotiation.cc:407] Connection header received
0808 18:32:35.716368 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:35.716372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:35.716427 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:35.716505 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:35.717647 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.718518 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.719792 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.720169 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.720258 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:35.722981 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:35.722984 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:35.722986 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:35.723010 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:35.724506 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:35.725024 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:35.725028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:35.725029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:35.725112 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:35.725492 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:35.725497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:35.725499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:35.725683 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:35.725792 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:35.726282 (+ 490us) server_negotiation.cc:299] Negotiation successful
0808 18:32:35.726441 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
I20260808 18:32:35.727567 16761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:35.714305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:41371 (local address 127.29.78.130:60335)
0808 18:32:35.714865 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:35.714907 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:35.716150 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:35.716688 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:35.716695 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:35.717034 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:32:35.717492 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:35.717505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.718646 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.718648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:35.719660 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:35.719669 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.720302 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.720305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:35.720475 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:35.721082 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:32:35.721097 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:32:35.722793 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:32:35.724621 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:32:35.724627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:32:35.724640 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:32:35.724892 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:32:35.725204 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:32:35.725208 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:32:35.725210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:32:35.725383 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:32:35.725791 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:32:35.725797 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:32:35.726044 (+ 247us) client_negotiation.cc:769] Sending connection context
0808 18:32:35.726237 (+ 193us) client_negotiation.cc:240] Negotiation successful
0808 18:32:35.726574 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":152,"threads_started":1}
I20260808 18:32:35.728811 16759 heartbeater.cc:347] Connected to a master server at 127.29.78.190:41371
I20260808 18:32:35.729077 16759 heartbeater.cc:464] Registering TS with master...
I20260808 18:32:35.729652 16759 heartbeater.cc:511] Master 127.29.78.190:41371 requested a full tablet report, sending...
I20260808 18:32:35.730880 16410 ts_manager.cc:194] Registered new tserver with Master: b64dba75c59c48d987967d03d113d4af (127.29.78.130:44271)
I20260808 18:32:35.731820 16410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:60335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260808 18:32:35.745092 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41371
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:35.857295 16764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:35.857581 16764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:35.857659 16764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:35.861666 16764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:35.861755 16764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:35.861845 16764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:32:35.867045 16764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:41371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:35.868129 16764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:35.869135 16764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:35.878813 16769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:35.878883 16770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:35.879078 16764 server_base.cc:1034] running on GCE node
W20260808 18:32:35.879158 16774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:35.879513 16764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:35.880206 16764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:35.881389 16764 hybrid_clock.cc:648] HybridClock initialized: now 1786213955881375 us; error 26 us; skew 500 ppm
Aug 08 18:32:35 dist-test-slave-gp6n krb5kdc[16360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:35.884086 16764 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:32:35.884966 16764 webserver.cc:460] Webserver started at http://127.29.78.131:42949/ using document root <none> and password file <none>
I20260808 18:32:35.885528 16764 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:35.885619 16764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:35.885855 16764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:35.887897 16764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "a31b71ec688f4519a0f93246ba01b755"
format_stamp: "Formatted at 2026-08-08 18:32:35 on dist-test-slave-gp6n"
server_key: "68981f5632743dcd7d5582acdb9937de"
server_key_iv: "7e9fdbf1c360251263019f5ec5782ed0"
server_key_version: "encryptionkey@0"
I20260808 18:32:35.888468 16764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "a31b71ec688f4519a0f93246ba01b755"
format_stamp: "Formatted at 2026-08-08 18:32:35 on dist-test-slave-gp6n"
server_key: "68981f5632743dcd7d5582acdb9937de"
server_key_iv: "7e9fdbf1c360251263019f5ec5782ed0"
server_key_version: "encryptionkey@0"
I20260808 18:32:35.892133 16764 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260808 18:32:35.894384 16783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:35.895618 16764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:32:35.895740 16764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "a31b71ec688f4519a0f93246ba01b755"
format_stamp: "Formatted at 2026-08-08 18:32:35 on dist-test-slave-gp6n"
server_key: "68981f5632743dcd7d5582acdb9937de"
server_key_iv: "7e9fdbf1c360251263019f5ec5782ed0"
server_key_version: "encryptionkey@0"
I20260808 18:32:35.895845 16764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:35.913627 16764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:35.916376 16764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:35.916581 16764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:35.917151 16764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:32:35.917948 16764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:32:35.918000 16764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:35.918072 16764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:32:35.918110 16764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:35.927163 16764 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:38149
I20260808 18:32:35.927191 16911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:38149 every 8 connection(s)
I20260808 18:32:35.928066 16764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:32:35.931597 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 16764
I20260808 18:32:35.931743 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:32:35.932101 30010 external_mini_cluster.cc:1428] Setting key 42b2357c185e17e7577fa886f1b31df4
Aug 08 18:32:35 dist-test-slave-gp6n krb5kdc[16360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:32:35.942075 16450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:35.929801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:59305 (local address 127.29.78.190:41371)
0808 18:32:35.929952 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:35.929956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:35.930870 (+ 914us) server_negotiation.cc:407] Connection header received
0808 18:32:35.932585 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:35.932591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:35.932655 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:35.932732 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:35.934188 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.934822 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.935937 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.936152 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.936234 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:35.938699 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:35.938702 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:35.938704 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:35.938725 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:35.940052 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:35.940570 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:35.940575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:35.940577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:35.940663 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:35.940996 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:35.940999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:35.941001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:35.941201 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:35.941309 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:35.941784 (+ 475us) server_negotiation.cc:299] Negotiation successful
0808 18:32:35.941906 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 18:32:35.942914 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:35.930219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:41371 (local address 127.29.78.131:59305)
0808 18:32:35.930716 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:35.930759 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:35.932318 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:35.932937 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:35.932963 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:35.933463 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:32:35.934026 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:35.934048 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.934932 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.934935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:35.935822 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:35.935831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.936278 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.936280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:35.936413 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:35.937026 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:32:35.937042 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:32:35.938558 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:32:35.940188 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:32:35.940194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:32:35.940207 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:32:35.940460 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:32:35.940754 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:32:35.940758 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:32:35.940760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:32:35.940898 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:32:35.941307 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:32:35.941313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:32:35.941552 (+ 239us) client_negotiation.cc:769] Sending connection context
0808 18:32:35.941739 (+ 187us) client_negotiation.cc:240] Negotiation successful
0808 18:32:35.942056 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":172,"threads_started":1}
I20260808 18:32:35.943958 16912 heartbeater.cc:347] Connected to a master server at 127.29.78.190:41371
I20260808 18:32:35.944203 16912 heartbeater.cc:464] Registering TS with master...
I20260808 18:32:35.944809 16912 heartbeater.cc:511] Master 127.29.78.190:41371 requested a full tablet report, sending...
I20260808 18:32:35.945920 16409 ts_manager.cc:194] Registered new tserver with Master: a31b71ec688f4519a0f93246ba01b755 (127.29.78.131:38149)
I20260808 18:32:35.946600 16409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:59305
I20260808 18:32:35.957702 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:32:35.967166 16450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:35.959794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46946 (local address 127.29.78.190:41371)
0808 18:32:35.959947 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:35.959951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:35.960189 (+ 238us) server_negotiation.cc:407] Connection header received
0808 18:32:35.960259 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:35.960261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:35.960300 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:35.960366 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:35.961173 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.961675 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.962611 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.962856 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.962956 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:35.964343 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:35.964346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:35.964347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:35.964365 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:35.965582 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:35.965957 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:35.965959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:35.965960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:35.966045 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:35.966276 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:35.966278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:35.966279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:35.966558 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:35.966662 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:35.966860 (+ 198us) server_negotiation.cc:299] Negotiation successful
0808 18:32:35.966952 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 18:32:35.969888 16409 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:32:35.972942 16409 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:32:35.987587 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:35.981979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36962 (local address 127.29.78.131:38149)
0808 18:32:35.982305 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:35.982318 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:35.982344 (+ 26us) server_negotiation.cc:407] Connection header received
0808 18:32:35.982491 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:35.982504 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:35.982722 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:35.982887 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:35.983237 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.984443 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.986130 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.986936 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.987073 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:35.987224 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:35.987348 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 18:32:35.987439 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":150,"threads_started":1}
I20260808 18:32:35.987587 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:35.981815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:38149 (local address 127.0.0.1:36962)
0808 18:32:35.982052 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:35.982066 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:35.982154 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:35.982873 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:35.982880 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:35.982900 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:35.983051 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:35.983060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.984591 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.984595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:35.986010 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:35.986019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.987062 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.987066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:35.987168 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:35.987204 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 18:32:35.987321 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 18:32:35.987439 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":47,"thread_start_us":108,"threads_started":1}
I20260808 18:32:35.987764 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:35.981269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44271 (local address 127.0.0.1:59220)
0808 18:32:35.981788 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:35.981801 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:35.982315 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:35.983736 (+ 1421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:35.983739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:35.983756 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:35.983857 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:35.983865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.984914 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.984918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:35.986040 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:35.986052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.987308 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.987311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:35.987411 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:35.987434 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 18:32:35.987525 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 18:32:35.987613 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":3,"thread_start_us":105,"threads_started":1}
I20260808 18:32:35.988011 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:35.981134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:42675 (local address 127.0.0.1:45012)
0808 18:32:35.981634 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:35.981683 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:35.981836 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:35.983212 (+ 1376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:35.983216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:35.983233 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:35.983316 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:35.983321 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.984656 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.984660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:35.986555 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:35.986565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.987658 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.987662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:35.987742 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:35.987763 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 18:32:35.987816 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 18:32:35.987880 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":457,"thread_start_us":63,"threads_started":1}
I20260808 18:32:35.988168 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:35.982474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59220 (local address 127.29.78.130:44271)
0808 18:32:35.983350 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:35.983364 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:35.983388 (+ 24us) server_negotiation.cc:407] Connection header received
0808 18:32:35.983456 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:35.983460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:35.983607 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:35.983757 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:35.983954 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.984776 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.986236 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.987012 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.987114 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:35.987240 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:35.987923 (+ 683us) server_negotiation.cc:299] Negotiation successful
0808 18:32:35.988013 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":170,"threads_started":1}
I20260808 18:32:35.988617 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:35.982121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.29.78.129:42675)
0808 18:32:35.982767 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:35.982786 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:35.982811 (+ 25us) server_negotiation.cc:407] Connection header received
0808 18:32:35.982886 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:35.982891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:35.983071 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:35.983244 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:35.983399 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.984483 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.986683 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:35.987535 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:35.988195 (+ 660us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:35.988341 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:35.988397 (+ 56us) server_negotiation.cc:299] Negotiation successful
0808 18:32:35.988486 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":73,"threads_started":1}
I20260808 18:32:35.991001 16547 tablet_service.cc:1467] Processing CreateTablet for tablet b8c0b81a852f4b9792ad7c9f233dd2f7 (DEFAULT_TABLE table=test-table [id=5a756e1ef1db4c418d326db4bc59ff05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:32:35.991034 16688 tablet_service.cc:1467] Processing CreateTablet for tablet b8c0b81a852f4b9792ad7c9f233dd2f7 (DEFAULT_TABLE table=test-table [id=5a756e1ef1db4c418d326db4bc59ff05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:32:35.991034 16836 tablet_service.cc:1467] Processing CreateTablet for tablet b8c0b81a852f4b9792ad7c9f233dd2f7 (DEFAULT_TABLE table=test-table [id=5a756e1ef1db4c418d326db4bc59ff05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:32:35.991954 16547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c0b81a852f4b9792ad7c9f233dd2f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:35.992035 16836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c0b81a852f4b9792ad7c9f233dd2f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:35.992035 16688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c0b81a852f4b9792ad7c9f233dd2f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:35.997764 16932 tablet_bootstrap.cc:492] T b8c0b81a852f4b9792ad7c9f233dd2f7 P bd99100fd60e4acbb2a98a310bf11514: Bootstrap starting.
I20260808 18:32:35.999676 16934 tablet_bootstrap.cc:492] T b8c0b81a852f4b9792ad7c9f233dd2f7 P a31b71ec688f4519a0f93246ba01b755: Bootstrap starting.
I20260808 18:32:36.000237 16933 tablet_bootstrap.cc:492] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af: Bootstrap starting.
I20260808 18:32:36.000465 16932 tablet_bootstrap.cc:654] T b8c0b81a852f4b9792ad7c9f233dd2f7 P bd99100fd60e4acbb2a98a310bf11514: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:36.001827 16932 log.cc:824] T b8c0b81a852f4b9792ad7c9f233dd2f7 P bd99100fd60e4acbb2a98a310bf11514: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:36.002045 16933 tablet_bootstrap.cc:654] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:36.002049 16934 tablet_bootstrap.cc:654] T b8c0b81a852f4b9792ad7c9f233dd2f7 P a31b71ec688f4519a0f93246ba01b755: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:36.003424 16934 log.cc:824] T b8c0b81a852f4b9792ad7c9f233dd2f7 P a31b71ec688f4519a0f93246ba01b755: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:36.003813 16933 log.cc:824] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:36.004113 16932 tablet_bootstrap.cc:492] T b8c0b81a852f4b9792ad7c9f233dd2f7 P bd99100fd60e4acbb2a98a310bf11514: No bootstrap required, opened a new log
I20260808 18:32:36.004379 16932 ts_tablet_manager.cc:1397] T b8c0b81a852f4b9792ad7c9f233dd2f7 P bd99100fd60e4acbb2a98a310bf11514: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260808 18:32:36.005342 16934 tablet_bootstrap.cc:492] T b8c0b81a852f4b9792ad7c9f233dd2f7 P a31b71ec688f4519a0f93246ba01b755: No bootstrap required, opened a new log
I20260808 18:32:36.005543 16934 ts_tablet_manager.cc:1397] T b8c0b81a852f4b9792ad7c9f233dd2f7 P a31b71ec688f4519a0f93246ba01b755: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:32:36.005704 16933 tablet_bootstrap.cc:492] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af: No bootstrap required, opened a new log
I20260808 18:32:36.005872 16933 ts_tablet_manager.cc:1397] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260808 18:32:36.007601 16932 raft_consensus.cc:348] T b8c0b81a852f4b9792ad7c9f233dd2f7 P bd99100fd60e4acbb2a98a310bf11514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd99100fd60e4acbb2a98a310bf11514" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42675 } } peers { permanent_uuid: "b64dba75c59c48d987967d03d113d4af" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44271 } } peers { permanent_uuid: "a31b71ec688f4519a0f93246ba01b755" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38149 } }
I20260808 18:32:36.007841 16932 raft_consensus.cc:374] T b8c0b81a852f4b9792ad7c9f233dd2f7 P bd99100fd60e4acbb2a98a310bf11514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:36.007915 16932 raft_consensus.cc:729] T b8c0b81a852f4b9792ad7c9f233dd2f7 P bd99100fd60e4acbb2a98a310bf11514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd99100fd60e4acbb2a98a310bf11514, State: Initialized, Role: FOLLOWER
I20260808 18:32:36.008453 16932 consensus_queue.cc:260] T b8c0b81a852f4b9792ad7c9f233dd2f7 P bd99100fd60e4acbb2a98a310bf11514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd99100fd60e4acbb2a98a310bf11514" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42675 } } peers { permanent_uuid: "b64dba75c59c48d987967d03d113d4af" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44271 } } peers { permanent_uuid: "a31b71ec688f4519a0f93246ba01b755" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38149 } }
I20260808 18:32:36.009336 16934 raft_consensus.cc:348] T b8c0b81a852f4b9792ad7c9f233dd2f7 P a31b71ec688f4519a0f93246ba01b755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd99100fd60e4acbb2a98a310bf11514" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42675 } } peers { permanent_uuid: "b64dba75c59c48d987967d03d113d4af" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44271 } } peers { permanent_uuid: "a31b71ec688f4519a0f93246ba01b755" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38149 } }
I20260808 18:32:36.009536 16932 ts_tablet_manager.cc:1428] T b8c0b81a852f4b9792ad7c9f233dd2f7 P bd99100fd60e4acbb2a98a310bf11514: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260808 18:32:36.009593 16934 raft_consensus.cc:374] T b8c0b81a852f4b9792ad7c9f233dd2f7 P a31b71ec688f4519a0f93246ba01b755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:36.009660 16934 raft_consensus.cc:729] T b8c0b81a852f4b9792ad7c9f233dd2f7 P a31b71ec688f4519a0f93246ba01b755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a31b71ec688f4519a0f93246ba01b755, State: Initialized, Role: FOLLOWER
I20260808 18:32:36.009611 16933 raft_consensus.cc:348] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd99100fd60e4acbb2a98a310bf11514" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42675 } } peers { permanent_uuid: "b64dba75c59c48d987967d03d113d4af" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44271 } } peers { permanent_uuid: "a31b71ec688f4519a0f93246ba01b755" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38149 } }
I20260808 18:32:36.009877 16933 raft_consensus.cc:374] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:36.009881 16615 heartbeater.cc:503] Master 127.29.78.190:41371 was elected leader, sending a full tablet report...
I20260808 18:32:36.009953 16933 raft_consensus.cc:729] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64dba75c59c48d987967d03d113d4af, State: Initialized, Role: FOLLOWER
I20260808 18:32:36.010084 16934 consensus_queue.cc:260] T b8c0b81a852f4b9792ad7c9f233dd2f7 P a31b71ec688f4519a0f93246ba01b755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd99100fd60e4acbb2a98a310bf11514" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42675 } } peers { permanent_uuid: "b64dba75c59c48d987967d03d113d4af" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44271 } } peers { permanent_uuid: "a31b71ec688f4519a0f93246ba01b755" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38149 } }
I20260808 18:32:36.010356 16933 consensus_queue.cc:260] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd99100fd60e4acbb2a98a310bf11514" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42675 } } peers { permanent_uuid: "b64dba75c59c48d987967d03d113d4af" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44271 } } peers { permanent_uuid: "a31b71ec688f4519a0f93246ba01b755" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38149 } }
I20260808 18:32:36.010998 16912 heartbeater.cc:503] Master 127.29.78.190:41371 was elected leader, sending a full tablet report...
I20260808 18:32:36.011034 16934 ts_tablet_manager.cc:1428] T b8c0b81a852f4b9792ad7c9f233dd2f7 P a31b71ec688f4519a0f93246ba01b755: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 18:32:36.011233 16933 ts_tablet_manager.cc:1428] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 18:32:36.011993 16759 heartbeater.cc:503] Master 127.29.78.190:41371 was elected leader, sending a full tablet report...
I20260808 18:32:36.105924 16940 raft_consensus.cc:482] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:32:36.106191 16940 raft_consensus.cc:504] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd99100fd60e4acbb2a98a310bf11514" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42675 } } peers { permanent_uuid: "b64dba75c59c48d987967d03d113d4af" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44271 } } peers { permanent_uuid: "a31b71ec688f4519a0f93246ba01b755" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38149 } }
I20260808 18:32:36.107517 16940 leader_election.cc:288] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd99100fd60e4acbb2a98a310bf11514 (127.29.78.129:42675), a31b71ec688f4519a0f93246ba01b755 (127.29.78.131:38149)
I20260808 18:32:36.112601 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:36.107907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:53087 (local address 127.29.78.129:42675)
0808 18:32:36.108047 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:36.108051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:36.108068 (+ 17us) server_negotiation.cc:407] Connection header received
0808 18:32:36.108218 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:36.108222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:36.108291 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:36.108400 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:36.108763 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:36.109571 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:36.111358 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:36.111960 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:36.112089 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:36.112143 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:36.112395 (+ 252us) server_negotiation.cc:299] Negotiation successful
0808 18:32:36.112469 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 18:32:36.112601 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:36.107964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:57117 (local address 127.29.78.131:38149)
0808 18:32:36.108107 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:36.108114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:36.108374 (+ 260us) server_negotiation.cc:407] Connection header received
0808 18:32:36.108480 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:36.108485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:36.108545 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:36.108609 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:36.108908 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:36.109655 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:36.111358 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:36.111960 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:36.112096 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:36.112143 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:36.112395 (+ 252us) server_negotiation.cc:299] Negotiation successful
0808 18:32:36.112469 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 18:32:36.112644 16761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:36.107765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:42675 (local address 127.29.78.130:53087)
0808 18:32:36.107952 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:36.107971 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:36.108077 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:36.108434 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:36.108440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:36.108461 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:36.108619 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:36.108627 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:36.109681 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:36.109684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:36.111224 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:36.111232 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:36.112091 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:36.112095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:36.112246 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:36.112287 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 18:32:36.112366 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 18:32:36.112476 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":165}
I20260808 18:32:36.112655 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:36.107765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:38149 (local address 127.29.78.130:57117)
0808 18:32:36.108267 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:36.108278 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:36.108361 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:36.108645 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:36.108650 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:36.108671 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:36.108798 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:36.108803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:36.109754 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:36.109757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:36.111221 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:36.111232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:36.112091 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:36.112095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:36.112241 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:36.112286 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 18:32:36.112366 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 18:32:36.112480 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":257,"thread_start_us":171,"threads_started":1}
I20260808 18:32:36.113416 16568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c0b81a852f4b9792ad7c9f233dd2f7" candidate_uuid: "b64dba75c59c48d987967d03d113d4af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd99100fd60e4acbb2a98a310bf11514" is_pre_election: true
I20260808 18:32:36.113583 16859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c0b81a852f4b9792ad7c9f233dd2f7" candidate_uuid: "b64dba75c59c48d987967d03d113d4af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a31b71ec688f4519a0f93246ba01b755" is_pre_election: true
I20260808 18:32:36.113752 16568 raft_consensus.cc:2454] T b8c0b81a852f4b9792ad7c9f233dd2f7 P bd99100fd60e4acbb2a98a310bf11514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b64dba75c59c48d987967d03d113d4af in term 0.
I20260808 18:32:36.113907 16859 raft_consensus.cc:2454] T b8c0b81a852f4b9792ad7c9f233dd2f7 P a31b71ec688f4519a0f93246ba01b755 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b64dba75c59c48d987967d03d113d4af in term 0.
I20260808 18:32:36.114310 16639 leader_election.cc:302] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b64dba75c59c48d987967d03d113d4af, bd99100fd60e4acbb2a98a310bf11514; no voters:
I20260808 18:32:36.114653 16940 raft_consensus.cc:2781] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:32:36.114742 16940 raft_consensus.cc:482] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:32:36.114786 16940 raft_consensus.cc:3037] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:36.115825 16940 raft_consensus.cc:504] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd99100fd60e4acbb2a98a310bf11514" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42675 } } peers { permanent_uuid: "b64dba75c59c48d987967d03d113d4af" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44271 } } peers { permanent_uuid: "a31b71ec688f4519a0f93246ba01b755" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38149 } }
I20260808 18:32:36.116178 16940 leader_election.cc:288] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af [CANDIDATE]: Term 1 election: Requested vote from peers bd99100fd60e4acbb2a98a310bf11514 (127.29.78.129:42675), a31b71ec688f4519a0f93246ba01b755 (127.29.78.131:38149)
I20260808 18:32:36.116490 16568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c0b81a852f4b9792ad7c9f233dd2f7" candidate_uuid: "b64dba75c59c48d987967d03d113d4af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd99100fd60e4acbb2a98a310bf11514"
I20260808 18:32:36.116554 16859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c0b81a852f4b9792ad7c9f233dd2f7" candidate_uuid: "b64dba75c59c48d987967d03d113d4af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a31b71ec688f4519a0f93246ba01b755"
I20260808 18:32:36.116642 16568 raft_consensus.cc:3037] T b8c0b81a852f4b9792ad7c9f233dd2f7 P bd99100fd60e4acbb2a98a310bf11514 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:36.116667 16859 raft_consensus.cc:3037] T b8c0b81a852f4b9792ad7c9f233dd2f7 P a31b71ec688f4519a0f93246ba01b755 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:36.117697 16568 raft_consensus.cc:2454] T b8c0b81a852f4b9792ad7c9f233dd2f7 P bd99100fd60e4acbb2a98a310bf11514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b64dba75c59c48d987967d03d113d4af in term 1.
I20260808 18:32:36.117714 16859 raft_consensus.cc:2454] T b8c0b81a852f4b9792ad7c9f233dd2f7 P a31b71ec688f4519a0f93246ba01b755 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b64dba75c59c48d987967d03d113d4af in term 1.
I20260808 18:32:36.118027 16640 leader_election.cc:302] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a31b71ec688f4519a0f93246ba01b755, b64dba75c59c48d987967d03d113d4af; no voters:
I20260808 18:32:36.118216 16940 raft_consensus.cc:2781] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:32:36.118472 16940 raft_consensus.cc:686] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af [term 1 LEADER]: Becoming Leader. State: Replica: b64dba75c59c48d987967d03d113d4af, State: Running, Role: LEADER
I20260808 18:32:36.118806 16940 consensus_queue.cc:237] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd99100fd60e4acbb2a98a310bf11514" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42675 } } peers { permanent_uuid: "b64dba75c59c48d987967d03d113d4af" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44271 } } peers { permanent_uuid: "a31b71ec688f4519a0f93246ba01b755" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38149 } }
I20260808 18:32:36.122148 16410 catalog_manager.cc:5495] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af reported cstate change: term changed from 0 to 1, leader changed from <none> to b64dba75c59c48d987967d03d113d4af (127.29.78.130). New cstate: current_term: 1 leader_uuid: "b64dba75c59c48d987967d03d113d4af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd99100fd60e4acbb2a98a310bf11514" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b64dba75c59c48d987967d03d113d4af" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a31b71ec688f4519a0f93246ba01b755" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38149 } health_report { overall_health: UNKNOWN } } }
I20260808 18:32:36.143913 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:36.140559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59222 (local address 127.29.78.130:44271)
0808 18:32:36.140686 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:36.140689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:36.140704 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:32:36.140854 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:36.140856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:36.140910 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:36.141012 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:32:36.141228 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:36.141725 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:36.142574 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:36.142774 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:36.142849 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:36.143104 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:32:36.143513 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:32:36.143603 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:36.143666 (+ 63us) server_negotiation.cc:299] Negotiation successful
0808 18:32:36.143711 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 18:32:36.149665 16568 raft_consensus.cc:1261] T b8c0b81a852f4b9792ad7c9f233dd2f7 P bd99100fd60e4acbb2a98a310bf11514 [term 1 FOLLOWER]: Refusing update from remote peer b64dba75c59c48d987967d03d113d4af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:32:36.149664 16859 raft_consensus.cc:1261] T b8c0b81a852f4b9792ad7c9f233dd2f7 P a31b71ec688f4519a0f93246ba01b755 [term 1 FOLLOWER]: Refusing update from remote peer b64dba75c59c48d987967d03d113d4af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:32:36.150343 16940 consensus_queue.cc:1035] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a31b71ec688f4519a0f93246ba01b755" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:32:36.150599 16942 consensus_queue.cc:1035] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd99100fd60e4acbb2a98a310bf11514" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:32:36.159543 16945 mvcc.cc:205] Tried to move back new op lower bound from 7316332364381986816 to 7316332364265242624. Current Snapshot: MvccSnapshot[applied={T|T < 7316332364381986816}]
I20260808 18:32:36.160609 16946 mvcc.cc:205] Tried to move back new op lower bound from 7316332364381986816 to 7316332364265242624. Current Snapshot: MvccSnapshot[applied={T|T < 7316332364381986816}]
I20260808 18:32:36.163683 16948 mvcc.cc:205] Tried to move back new op lower bound from 7316332364381986816 to 7316332364265242624. Current Snapshot: MvccSnapshot[applied={T|T < 7316332364381986816}]
I20260808 18:32:36.167949 16409 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:32:36.168139 16409 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:32:36.170723 16409 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a907c74ebe024aec85f636f2de05f69b: Sending DeleteTablet for 3 replicas of tablet b8c0b81a852f4b9792ad7c9f233dd2f7
I20260808 18:32:36.171478 16547 tablet_service.cc:1514] Processing DeleteTablet for tablet b8c0b81a852f4b9792ad7c9f233dd2f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:32:36 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:45012
I20260808 18:32:36.171571 16836 tablet_service.cc:1514] Processing DeleteTablet for tablet b8c0b81a852f4b9792ad7c9f233dd2f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:32:36 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:36962
I20260808 18:32:36.171744 16688 tablet_service.cc:1514] Processing DeleteTablet for tablet b8c0b81a852f4b9792ad7c9f233dd2f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:32:36 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:59220
I20260808 18:32:36.171988 16954 tablet_replica.cc:330] T b8c0b81a852f4b9792ad7c9f233dd2f7 P bd99100fd60e4acbb2a98a310bf11514: stopping tablet replica
I20260808 18:32:36.172333 16955 tablet_replica.cc:330] T b8c0b81a852f4b9792ad7c9f233dd2f7 P a31b71ec688f4519a0f93246ba01b755: stopping tablet replica
I20260808 18:32:36.172622 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 16477
I20260808 18:32:36.172755 16955 raft_consensus.cc:2229] T b8c0b81a852f4b9792ad7c9f233dd2f7 P a31b71ec688f4519a0f93246ba01b755 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:32:36.173097 16955 raft_consensus.cc:2258] T b8c0b81a852f4b9792ad7c9f233dd2f7 P a31b71ec688f4519a0f93246ba01b755 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:32:36.173382 16956 tablet_replica.cc:330] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af: stopping tablet replica
I20260808 18:32:36.173739 16956 raft_consensus.cc:2229] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:32:36.174160 16956 raft_consensus.cc:2258] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:32:36.177179 16956 ts_tablet_manager.cc:1905] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:32:36.179625 16956 ts_tablet_manager.cc:1918] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:32:36.179742 16956 log.cc:1196] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/b8c0b81a852f4b9792ad7c9f233dd2f7
I20260808 18:32:36.180096 16956 ts_tablet_manager.cc:1939] T b8c0b81a852f4b9792ad7c9f233dd2f7 P b64dba75c59c48d987967d03d113d4af: Deleting consensus metadata
I20260808 18:32:36.181289 16394 catalog_manager.cc:4841] TS b64dba75c59c48d987967d03d113d4af (127.29.78.130:44271): tablet b8c0b81a852f4b9792ad7c9f233dd2f7 (table test-table [id=5a756e1ef1db4c418d326db4bc59ff05]) successfully deleted
I20260808 18:32:36.183543 16955 ts_tablet_manager.cc:1905] T b8c0b81a852f4b9792ad7c9f233dd2f7 P a31b71ec688f4519a0f93246ba01b755: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 18:32:36.184363 16393 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:42675 (error 108)
I20260808 18:32:36.185242 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 16621
I20260808 18:32:36.185492 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:36.185153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:42675 (local address 127.0.0.1:45020)
0808 18:32:36.185333 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:36.185409 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:42675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260808 18:32:36.187601 16393 catalog_manager.cc:4584] TS bd99100fd60e4acbb2a98a310bf11514 (127.29.78.129:42675): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b8c0b81a852f4b9792ad7c9f233dd2f7: Network error: Client connection negotiation failed: client connection to 127.29.78.129:42675: connect: Connection refused (error 111)
I20260808 18:32:36.188706 16955 ts_tablet_manager.cc:1918] T b8c0b81a852f4b9792ad7c9f233dd2f7 P a31b71ec688f4519a0f93246ba01b755: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:32:36.188791 16955 log.cc:1196] T b8c0b81a852f4b9792ad7c9f233dd2f7 P a31b71ec688f4519a0f93246ba01b755: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/b8c0b81a852f4b9792ad7c9f233dd2f7
I20260808 18:32:36.189106 16955 ts_tablet_manager.cc:1939] T b8c0b81a852f4b9792ad7c9f233dd2f7 P a31b71ec688f4519a0f93246ba01b755: Deleting consensus metadata
I20260808 18:32:36.190209 16394 catalog_manager.cc:4841] TS a31b71ec688f4519a0f93246ba01b755 (127.29.78.131:38149): tablet b8c0b81a852f4b9792ad7c9f233dd2f7 (table test-table [id=5a756e1ef1db4c418d326db4bc59ff05]) successfully deleted
I20260808 18:32:36.193025 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 16764
I20260808 18:32:36.199597 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 16374
2026-08-08T18:32:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2444 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-08T18:32:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:32:36Z Disabled control of system clock
I20260808 18:32:36.233439 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:39149
--webserver_interface=127.29.78.190
--webserver_port=0
--builtin_ntp_servers=127.29.78.148:39595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:39149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 18:32:36.345506 16961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:36.345769 16961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:36.345805 16961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:36.349892 16961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:32:36.349952 16961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:36.349968 16961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:36.349985 16961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:32:36.350001 16961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:32:36.354837 16961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:39149
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:39149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:36.355875 16961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:36.357228 16961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:36.366004 16966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:36.366114 16961 server_base.cc:1034] running on GCE node
W20260808 18:32:36.365975 16967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:36.365975 16969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:36.367029 16961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:36.368044 16961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:36.369204 16961 hybrid_clock.cc:648] HybridClock initialized: now 1786213956369185 us; error 32 us; skew 500 ppm
I20260808 18:32:36.371122 16961 webserver.cc:460] Webserver started at http://127.29.78.190:41517/ using document root <none> and password file <none>
I20260808 18:32:36.371727 16961 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:36.371795 16961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:36.372033 16961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:36.374094 16961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "3f72bf1265ce40629e4c33681174e6e7"
format_stamp: "Formatted at 2026-08-08 18:32:36 on dist-test-slave-gp6n"
server_key: "35d445b91195b491b9971c53be683674"
server_key_iv: "dc3e810c7d70f94089d64aa16d98b474"
server_key_version: "encryptionkey@0"
I20260808 18:32:36.374696 16961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "3f72bf1265ce40629e4c33681174e6e7"
format_stamp: "Formatted at 2026-08-08 18:32:36 on dist-test-slave-gp6n"
server_key: "35d445b91195b491b9971c53be683674"
server_key_iv: "dc3e810c7d70f94089d64aa16d98b474"
server_key_version: "encryptionkey@0"
I20260808 18:32:36.378511 16961 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 18:32:36.380832 16976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:36.382061 16961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:32:36.382192 16961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "3f72bf1265ce40629e4c33681174e6e7"
format_stamp: "Formatted at 2026-08-08 18:32:36 on dist-test-slave-gp6n"
server_key: "35d445b91195b491b9971c53be683674"
server_key_iv: "dc3e810c7d70f94089d64aa16d98b474"
server_key_version: "encryptionkey@0"
I20260808 18:32:36.382294 16961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:36.401018 16961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:36.401763 16961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:36.401976 16961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:36.409096 16961 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:39149
I20260808 18:32:36.409114 17032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:39149 every 8 connection(s)
I20260808 18:32:36.410127 16961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:32:36.413226 17033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:36.418879 17033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7: Bootstrap starting.
I20260808 18:32:36.419577 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 16961
I20260808 18:32:36.419710 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:32:36.420025 30010 external_mini_cluster.cc:1428] Setting key 1ffe6f933bbf9ebb93bd367994421c5e
I20260808 18:32:36.421927 17033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:36.422762 17033 log.cc:824] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:36.424707 17033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7: No bootstrap required, opened a new log
I20260808 18:32:36.428351 17033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f72bf1265ce40629e4c33681174e6e7" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39149 } }
I20260808 18:32:36.428570 17033 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:36.428645 17033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f72bf1265ce40629e4c33681174e6e7, State: Initialized, Role: FOLLOWER
I20260808 18:32:36.429116 17033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f72bf1265ce40629e4c33681174e6e7" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39149 } }
I20260808 18:32:36.429317 17033 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:32:36.429404 17033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:32:36.429534 17033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:36.430255 17036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:36.421380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47186 (local address 127.29.78.190:39149)
0808 18:32:36.422104 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:36.422128 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:36.422167 (+ 39us) server_negotiation.cc:407] Connection header received
0808 18:32:36.422901 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:36.422932 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:36.423418 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:36.423775 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:36.424232 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:36.425456 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:36.426490 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:36.427038 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:36.427228 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:36.428175 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:36.428180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:36.428193 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:36.428680 (+ 487us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:36.428787 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:36.428811 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:36.428925 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:36.429091 (+ 166us) server_negotiation.cc:299] Negotiation successful
0808 18:32:36.429404 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":125,"threads_started":1}
I20260808 18:32:36.430485 17033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f72bf1265ce40629e4c33681174e6e7" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39149 } }
I20260808 18:32:36.430857 17033 leader_election.cc:302] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f72bf1265ce40629e4c33681174e6e7; no voters:
I20260808 18:32:36.431208 17033 leader_election.cc:288] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:32:36.431281 17039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:32:36.432476 17033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:32:36.432545 17039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7 [term 1 LEADER]: Becoming Leader. State: Replica: 3f72bf1265ce40629e4c33681174e6e7, State: Running, Role: LEADER
I20260808 18:32:36.432853 17039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f72bf1265ce40629e4c33681174e6e7" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39149 } }
I20260808 18:32:36.434726 17040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f72bf1265ce40629e4c33681174e6e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f72bf1265ce40629e4c33681174e6e7" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39149 } } }
I20260808 18:32:36.435437 17040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7 [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:36.434748 17041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f72bf1265ce40629e4c33681174e6e7. Latest consensus state: current_term: 1 leader_uuid: "3f72bf1265ce40629e4c33681174e6e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f72bf1265ce40629e4c33681174e6e7" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39149 } } }
I20260808 18:32:36.435616 17041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7 [sys.catalog]: This master's current role is: LEADER
W20260808 18:32:36.437994 17054 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:32:36.438069 17054 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:32:36.438169 17055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:32:36.438910 17055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:32:36.443912 17055 catalog_manager.cc:1347] Generated new cluster ID: f3ac355aed154f5386046a603475e803
I20260808 18:32:36.443979 17055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:32:36.462265 17055 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:32:36.463586 17055 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:32:36.470059 17055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7: Generated new TSK 0
I20260808 18:32:36.470867 17055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:32:36.476481 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39149
--builtin_ntp_servers=127.29.78.148:39595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 18:32:36.589867 17059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:36.590122 17059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:36.590161 17059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:36.594091 17059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:36.594143 17059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:36.594195 17059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:32:36.598753 17059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:36.599682 17059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:36.600975 17059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:36.610551 17066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:36.610662 17068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:36.610740 17059 server_base.cc:1034] running on GCE node
W20260808 18:32:36.610662 17065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:36.611228 17059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:36.611845 17059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:36.613010 17059 hybrid_clock.cc:648] HybridClock initialized: now 1786213956612998 us; error 25 us; skew 500 ppm
I20260808 18:32:36.614955 17059 webserver.cc:460] Webserver started at http://127.29.78.129:37707/ using document root <none> and password file <none>
I20260808 18:32:36.615523 17059 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:36.615595 17059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:36.615832 17059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:36.617848 17059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "84a5aca1a73948028d2a2b1bfc5bb0b8"
format_stamp: "Formatted at 2026-08-08 18:32:36 on dist-test-slave-gp6n"
server_key: "e9416332c268aed4054eb5d67aeb5a5a"
server_key_iv: "7d6a19941f878ded24c6090b3e5edbe2"
server_key_version: "encryptionkey@0"
I20260808 18:32:36.618467 17059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "84a5aca1a73948028d2a2b1bfc5bb0b8"
format_stamp: "Formatted at 2026-08-08 18:32:36 on dist-test-slave-gp6n"
server_key: "e9416332c268aed4054eb5d67aeb5a5a"
server_key_iv: "7d6a19941f878ded24c6090b3e5edbe2"
server_key_version: "encryptionkey@0"
I20260808 18:32:36.621958 17059 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:32:36.624365 17076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:36.625553 17059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:32:36.625687 17059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "84a5aca1a73948028d2a2b1bfc5bb0b8"
format_stamp: "Formatted at 2026-08-08 18:32:36 on dist-test-slave-gp6n"
server_key: "e9416332c268aed4054eb5d67aeb5a5a"
server_key_iv: "7d6a19941f878ded24c6090b3e5edbe2"
server_key_version: "encryptionkey@0"
I20260808 18:32:36.625798 17059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:36.642282 17059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:36.642982 17059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:36.643200 17059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:36.643744 17059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:32:36.644654 17059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:32:36.644703 17059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:36.644742 17059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:32:36.644758 17059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:36.653402 17059 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:44147
I20260808 18:32:36.653450 17198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:44147 every 8 connection(s)
I20260808 18:32:36.654331 17059 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:32:36.663452 17036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:36.656235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:45143 (local address 127.29.78.190:39149)
0808 18:32:36.656382 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:36.656386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:36.657467 (+ 1081us) server_negotiation.cc:407] Connection header received
0808 18:32:36.658882 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:36.658885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:36.658934 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:36.659017 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:36.659912 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:36.660455 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:36.661548 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:36.661762 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:36.661836 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:36.662959 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:36.662962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:36.662964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:36.662974 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:36.662997 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:36.663004 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:36.663060 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:36.663205 (+ 145us) server_negotiation.cc:299] Negotiation successful
0808 18:32:36.663254 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 18:32:36.663625 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 17059
I20260808 18:32:36.663759 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:32:36.664077 30010 external_mini_cluster.cc:1428] Setting key c36b4918e84284fe2f649ffc50c17070
I20260808 18:32:36.664314 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:36.656630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:39149 (local address 127.29.78.129:45143)
0808 18:32:36.657288 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:36.657331 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:36.658681 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:36.659184 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:36.659191 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:36.659274 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:32:36.659781 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:36.659794 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:36.660562 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:36.660564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:36.661438 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:36.661447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:36.661887 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:36.661889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:36.662022 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:36.662733 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:32:36.662735 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:32:36.662774 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:32:36.662834 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:32:36.663084 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:32:36.663091 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:32:36.663096 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 18:32:36.663188 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 18:32:36.663395 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":118,"threads_started":1}
I20260808 18:32:36.665622 17199 heartbeater.cc:347] Connected to a master server at 127.29.78.190:39149
I20260808 18:32:36.665850 17199 heartbeater.cc:464] Registering TS with master...
I20260808 18:32:36.666198 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39149
--builtin_ntp_servers=127.29.78.148:39595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 18:32:36.666446 17199 heartbeater.cc:511] Master 127.29.78.190:39149 requested a full tablet report, sending...
I20260808 18:32:36.667949 16995 ts_manager.cc:194] Registered new tserver with Master: 84a5aca1a73948028d2a2b1bfc5bb0b8 (127.29.78.129:44147)
I20260808 18:32:36.669595 16995 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.129:45143
W20260808 18:32:36.777869 17204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:36.778132 17204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:36.778170 17204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:36.782130 17204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:36.782188 17204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:36.782241 17204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:32:36.786988 17204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:36.788007 17204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:36.789364 17204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:36.798715 17210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:36.798725 17209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:36.798735 17212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:36.798939 17204 server_base.cc:1034] running on GCE node
I20260808 18:32:36.799376 17204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:36.799976 17204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:36.801131 17204 hybrid_clock.cc:648] HybridClock initialized: now 1786213956801098 us; error 48 us; skew 500 ppm
I20260808 18:32:36.802909 17204 webserver.cc:460] Webserver started at http://127.29.78.130:46597/ using document root <none> and password file <none>
I20260808 18:32:36.803424 17204 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:36.803490 17204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:36.803651 17204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:36.805543 17204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "0e33afc7b1634128b852099aa8eb6881"
format_stamp: "Formatted at 2026-08-08 18:32:36 on dist-test-slave-gp6n"
server_key: "60cedd6f625ff3acda4587c233358797"
server_key_iv: "3fc8c6f4001fd2f586558be2a29406a7"
server_key_version: "encryptionkey@0"
I20260808 18:32:36.806038 17204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "0e33afc7b1634128b852099aa8eb6881"
format_stamp: "Formatted at 2026-08-08 18:32:36 on dist-test-slave-gp6n"
server_key: "60cedd6f625ff3acda4587c233358797"
server_key_iv: "3fc8c6f4001fd2f586558be2a29406a7"
server_key_version: "encryptionkey@0"
I20260808 18:32:36.809695 17204 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 18:32:36.812036 17218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:36.813222 17204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:32:36.813357 17204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "0e33afc7b1634128b852099aa8eb6881"
format_stamp: "Formatted at 2026-08-08 18:32:36 on dist-test-slave-gp6n"
server_key: "60cedd6f625ff3acda4587c233358797"
server_key_iv: "3fc8c6f4001fd2f586558be2a29406a7"
server_key_version: "encryptionkey@0"
I20260808 18:32:36.813472 17204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:36.824719 17204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:36.825348 17204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:36.825547 17204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:36.826110 17204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:32:36.827128 17204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:32:36.827200 17204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:36.827272 17204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:32:36.827316 17204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:36.836763 17204 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:46621
I20260808 18:32:36.836781 17339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:46621 every 8 connection(s)
I20260808 18:32:36.837694 17204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:32:36.844091 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 17204
I20260808 18:32:36.844225 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:32:36.844576 30010 external_mini_cluster.cc:1428] Setting key 4ae4f7454875d986f06fade8191fadbd
I20260808 18:32:36.846920 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39149
--builtin_ntp_servers=127.29.78.148:39595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 18:32:36.847669 17036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:36.839507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:36749 (local address 127.29.78.190:39149)
0808 18:32:36.839665 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:36.839669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:36.840717 (+ 1048us) server_negotiation.cc:407] Connection header received
0808 18:32:36.842122 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:36.842125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:36.842175 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:36.842255 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:36.843233 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:36.843792 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:36.845369 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:36.845609 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:36.845728 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:36.847099 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:36.847105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:36.847109 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:36.847127 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:36.847158 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:36.847172 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:36.847264 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:36.847470 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 18:32:36.847536 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 18:32:36.848893 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:36.839907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:39149 (local address 127.29.78.130:36749)
0808 18:32:36.840566 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:36.840608 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:36.841924 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:36.842497 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:36.842507 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:36.842602 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:32:36.843102 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:36.843115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:36.843922 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:36.843925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:36.845233 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:36.845243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:36.845770 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:36.845774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:36.845954 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:36.846671 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:32:36.846675 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:32:36.846736 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:32:36.846833 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:32:36.847270 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:32:36.847283 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:32:36.847292 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 18:32:36.847432 (+ 140us) client_negotiation.cc:240] Negotiation successful
0808 18:32:36.847772 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":119,"threads_started":1}
I20260808 18:32:36.850272 17341 heartbeater.cc:347] Connected to a master server at 127.29.78.190:39149
I20260808 18:32:36.850550 17341 heartbeater.cc:464] Registering TS with master...
I20260808 18:32:36.851089 17341 heartbeater.cc:511] Master 127.29.78.190:39149 requested a full tablet report, sending...
I20260808 18:32:36.852231 16995 ts_manager.cc:194] Registered new tserver with Master: 0e33afc7b1634128b852099aa8eb6881 (127.29.78.130:46621)
I20260808 18:32:36.852823 16995 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.130:36749
W20260808 18:32:36.962798 17344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:36.963104 17344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:36.963176 17344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:36.967141 17344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:36.967223 17344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:36.967314 17344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:32:36.972328 17344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:36.973381 17344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:36.974808 17344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:36.984722 17349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:36.984740 17352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:36.984750 17350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:36.984975 17344 server_base.cc:1034] running on GCE node
I20260808 18:32:36.985460 17344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:36.986086 17344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:36.987244 17344 hybrid_clock.cc:648] HybridClock initialized: now 1786213956987199 us; error 57 us; skew 500 ppm
I20260808 18:32:36.989075 17344 webserver.cc:460] Webserver started at http://127.29.78.131:45777/ using document root <none> and password file <none>
I20260808 18:32:36.989656 17344 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:36.989719 17344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:36.989971 17344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:36.992036 17344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "e7abb819105d41488715d1f3a2443227"
format_stamp: "Formatted at 2026-08-08 18:32:36 on dist-test-slave-gp6n"
server_key: "0c04280f7c72906ce1889e4e8d4a3314"
server_key_iv: "e87e0c926bd9203660c9c1db7d31a64c"
server_key_version: "encryptionkey@0"
I20260808 18:32:36.992619 17344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "e7abb819105d41488715d1f3a2443227"
format_stamp: "Formatted at 2026-08-08 18:32:36 on dist-test-slave-gp6n"
server_key: "0c04280f7c72906ce1889e4e8d4a3314"
server_key_iv: "e87e0c926bd9203660c9c1db7d31a64c"
server_key_version: "encryptionkey@0"
I20260808 18:32:36.996392 17344 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:32:36.998718 17361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:36.999886 17344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:32:37.000021 17344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "e7abb819105d41488715d1f3a2443227"
format_stamp: "Formatted at 2026-08-08 18:32:36 on dist-test-slave-gp6n"
server_key: "0c04280f7c72906ce1889e4e8d4a3314"
server_key_iv: "e87e0c926bd9203660c9c1db7d31a64c"
server_key_version: "encryptionkey@0"
I20260808 18:32:37.000137 17344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:37.016845 17344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:37.017544 17344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:37.017745 17344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:37.018342 17344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:32:37.019297 17344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:32:37.019369 17344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:37.019452 17344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:32:37.019490 17344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:37.028272 17344 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:39837
I20260808 18:32:37.028312 17484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:39837 every 8 connection(s)
I20260808 18:32:37.029275 17344 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:32:37.034562 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 17344
I20260808 18:32:37.034691 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:32:37.034997 30010 external_mini_cluster.cc:1428] Setting key 262e02255658ba46cba2b464a760193e
I20260808 18:32:37.038815 17036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:37.031257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:54403 (local address 127.29.78.190:39149)
0808 18:32:37.031410 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:37.031413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:37.032285 (+ 872us) server_negotiation.cc:407] Connection header received
0808 18:32:37.033802 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:37.033805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:37.033891 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:37.033982 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:37.035062 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.035618 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.036771 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.037001 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.037121 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:37.038291 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:37.038296 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:37.038304 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:37.038315 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:37.038334 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:37.038341 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:37.038455 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:37.038618 (+ 163us) server_negotiation.cc:299] Negotiation successful
0808 18:32:37.038678 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 18:32:37.039638 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:37.031470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:39149 (local address 127.29.78.131:54403)
0808 18:32:37.032120 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:37.032161 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:37.033583 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:37.034175 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:37.034182 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:37.034268 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:32:37.034904 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:37.034920 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.035749 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.035754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:37.036638 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:37.036646 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.037157 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.037161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:37.037332 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:37.038033 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:32:37.038034 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:32:37.038076 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:32:37.038137 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:32:37.038447 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:32:37.038456 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:32:37.038463 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 18:32:37.038606 (+ 143us) client_negotiation.cc:240] Negotiation successful
0808 18:32:37.038883 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":105,"threads_started":1}
I20260808 18:32:37.040661 17486 heartbeater.cc:347] Connected to a master server at 127.29.78.190:39149
I20260808 18:32:37.040896 17486 heartbeater.cc:464] Registering TS with master...
I20260808 18:32:37.041410 17486 heartbeater.cc:511] Master 127.29.78.190:39149 requested a full tablet report, sending...
I20260808 18:32:37.042444 16995 ts_manager.cc:194] Registered new tserver with Master: e7abb819105d41488715d1f3a2443227 (127.29.78.131:39837)
I20260808 18:32:37.043033 16995 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.131:54403
I20260808 18:32:37.048828 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:32:37.054335 17036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:37.050851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47202 (local address 127.29.78.190:39149)
0808 18:32:37.051033 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:37.051037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:37.051133 (+ 96us) server_negotiation.cc:407] Connection header received
0808 18:32:37.051233 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:37.051235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:37.051276 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:37.051345 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:37.051597 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.052098 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.052873 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.053076 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.053151 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:37.053886 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:37.053889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:37.053891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:37.053903 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:37.053923 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:37.053931 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:37.053988 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:37.054140 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 18:32:37.054192 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 18:32:37.057032 16995 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:32:37.060441 16995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:32:37.075801 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:37.069497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51536 (local address 127.29.78.129:44147)
0808 18:32:37.070082 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:37.070097 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:37.070375 (+ 278us) server_negotiation.cc:407] Connection header received
0808 18:32:37.070723 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:37.070730 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:37.070935 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:37.071068 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:37.071719 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.072482 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.074341 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.075000 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.075110 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:37.075299 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:37.075484 (+ 185us) server_negotiation.cc:299] Negotiation successful
0808 18:32:37.075615 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":112,"threads_started":1}
I20260808 18:32:37.075910 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:37.069574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53914 (local address 127.29.78.131:39837)
0808 18:32:37.070280 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:37.070293 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:37.070321 (+ 28us) server_negotiation.cc:407] Connection header received
0808 18:32:37.070765 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:37.070769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:37.070935 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:37.071068 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:37.071752 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.072472 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.074313 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.075165 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.075264 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:37.075444 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:37.075657 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 18:32:37.075754 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":162,"threads_started":1}
I20260808 18:32:37.076287 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:37.069205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:39837 (local address 127.0.0.1:53914)
0808 18:32:37.070048 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:37.070100 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:37.070612 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:37.071274 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:37.071280 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:37.071320 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:37.071581 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:37.071590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.072661 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.072664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:37.074179 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:37.074187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.075381 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.075385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:37.075505 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:37.075529 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 18:32:37.075613 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 18:32:37.075739 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":47,"spinlock_wait_cycles":12416,"thread_start_us":91,"threads_started":1}
I20260808 18:32:37.076478 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:37.069962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:46621 (local address 127.0.0.1:43886)
0808 18:32:37.070580 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:37.070595 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:37.070692 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:37.071275 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:37.071280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:37.071324 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:37.071591 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:37.071598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.073744 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.073750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:37.074822 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:37.074833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.076079 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.076083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:37.076171 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:37.076192 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 18:32:37.076264 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 18:32:37.076338 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":179,"spinlock_wait_cycles":15744,"thread_start_us":111,"threads_started":1}
I20260808 18:32:37.076642 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:37.069291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:44147 (local address 127.0.0.1:51536)
0808 18:32:37.070281 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:37.070294 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:37.070519 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:37.071274 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:37.071280 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:37.071322 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:37.071587 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:37.071593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.072600 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.072603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:37.074176 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:37.074187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.075126 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.075130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:37.075254 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:37.075297 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 18:32:37.076350 (+ 1053us) client_negotiation.cc:240] Negotiation successful
0808 18:32:37.076442 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"mutex_wait_us":267,"spinlock_wait_cycles":7424,"thread_start_us":96,"threads_started":1}
I20260808 18:32:37.077234 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:37.070425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.29.78.130:46621)
0808 18:32:37.070879 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:37.070891 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:37.070908 (+ 17us) server_negotiation.cc:407] Connection header received
0808 18:32:37.070967 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:37.070971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:37.071118 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:37.071272 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:37.072652 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.073635 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.074963 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.075735 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.075835 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:37.076019 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:37.076989 (+ 970us) server_negotiation.cc:299] Negotiation successful
0808 18:32:37.077084 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":123,"threads_started":1}
I20260808 18:32:37.079675 17268 tablet_service.cc:1467] Processing CreateTablet for tablet ce7e307a2a1a40ccb3a1e0f91fbf73a8 (DEFAULT_TABLE table=test-table [id=2f2e606e4ae4498ea5ede3582747733a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:32:37.079676 17414 tablet_service.cc:1467] Processing CreateTablet for tablet ce7e307a2a1a40ccb3a1e0f91fbf73a8 (DEFAULT_TABLE table=test-table [id=2f2e606e4ae4498ea5ede3582747733a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:32:37.079676 17126 tablet_service.cc:1467] Processing CreateTablet for tablet ce7e307a2a1a40ccb3a1e0f91fbf73a8 (DEFAULT_TABLE table=test-table [id=2f2e606e4ae4498ea5ede3582747733a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:32:37.080638 17268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce7e307a2a1a40ccb3a1e0f91fbf73a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:37.080688 17414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce7e307a2a1a40ccb3a1e0f91fbf73a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:37.080688 17126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce7e307a2a1a40ccb3a1e0f91fbf73a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:37.086877 17504 tablet_bootstrap.cc:492] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881: Bootstrap starting.
I20260808 18:32:37.088330 17505 tablet_bootstrap.cc:492] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227: Bootstrap starting.
I20260808 18:32:37.088330 17507 tablet_bootstrap.cc:492] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 84a5aca1a73948028d2a2b1bfc5bb0b8: Bootstrap starting.
I20260808 18:32:37.089483 17504 tablet_bootstrap.cc:654] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:37.090225 17504 log.cc:824] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:37.090567 17507 tablet_bootstrap.cc:654] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 84a5aca1a73948028d2a2b1bfc5bb0b8: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:37.090567 17505 tablet_bootstrap.cc:654] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:37.091421 17505 log.cc:824] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:37.091421 17507 log.cc:824] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 84a5aca1a73948028d2a2b1bfc5bb0b8: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:37.092352 17504 tablet_bootstrap.cc:492] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881: No bootstrap required, opened a new log
I20260808 18:32:37.092557 17504 ts_tablet_manager.cc:1397] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:32:37.093367 17505 tablet_bootstrap.cc:492] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227: No bootstrap required, opened a new log
I20260808 18:32:37.093367 17507 tablet_bootstrap.cc:492] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 84a5aca1a73948028d2a2b1bfc5bb0b8: No bootstrap required, opened a new log
I20260808 18:32:37.093549 17507 ts_tablet_manager.cc:1397] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 84a5aca1a73948028d2a2b1bfc5bb0b8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260808 18:32:37.093549 17505 ts_tablet_manager.cc:1397] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260808 18:32:37.096148 17504 raft_consensus.cc:348] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7abb819105d41488715d1f3a2443227" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39837 } } peers { permanent_uuid: "84a5aca1a73948028d2a2b1bfc5bb0b8" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44147 } } peers { permanent_uuid: "0e33afc7b1634128b852099aa8eb6881" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46621 } }
I20260808 18:32:37.096395 17504 raft_consensus.cc:374] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:37.096457 17504 raft_consensus.cc:729] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e33afc7b1634128b852099aa8eb6881, State: Initialized, Role: FOLLOWER
I20260808 18:32:37.096902 17504 consensus_queue.cc:260] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7abb819105d41488715d1f3a2443227" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39837 } } peers { permanent_uuid: "84a5aca1a73948028d2a2b1bfc5bb0b8" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44147 } } peers { permanent_uuid: "0e33afc7b1634128b852099aa8eb6881" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46621 } }
I20260808 18:32:37.097568 17341 heartbeater.cc:503] Master 127.29.78.190:39149 was elected leader, sending a full tablet report...
I20260808 18:32:37.097923 17504 ts_tablet_manager.cc:1428] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260808 18:32:37.097826 17507 raft_consensus.cc:348] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 84a5aca1a73948028d2a2b1bfc5bb0b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7abb819105d41488715d1f3a2443227" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39837 } } peers { permanent_uuid: "84a5aca1a73948028d2a2b1bfc5bb0b8" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44147 } } peers { permanent_uuid: "0e33afc7b1634128b852099aa8eb6881" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46621 } }
I20260808 18:32:37.097826 17505 raft_consensus.cc:348] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7abb819105d41488715d1f3a2443227" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39837 } } peers { permanent_uuid: "84a5aca1a73948028d2a2b1bfc5bb0b8" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44147 } } peers { permanent_uuid: "0e33afc7b1634128b852099aa8eb6881" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46621 } }
I20260808 18:32:37.098069 17505 raft_consensus.cc:374] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:37.098069 17507 raft_consensus.cc:374] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 84a5aca1a73948028d2a2b1bfc5bb0b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:37.098121 17507 raft_consensus.cc:729] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 84a5aca1a73948028d2a2b1bfc5bb0b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84a5aca1a73948028d2a2b1bfc5bb0b8, State: Initialized, Role: FOLLOWER
I20260808 18:32:37.098121 17505 raft_consensus.cc:729] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7abb819105d41488715d1f3a2443227, State: Initialized, Role: FOLLOWER
I20260808 18:32:37.098567 17507 consensus_queue.cc:260] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 84a5aca1a73948028d2a2b1bfc5bb0b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7abb819105d41488715d1f3a2443227" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39837 } } peers { permanent_uuid: "84a5aca1a73948028d2a2b1bfc5bb0b8" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44147 } } peers { permanent_uuid: "0e33afc7b1634128b852099aa8eb6881" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46621 } }
I20260808 18:32:37.098567 17505 consensus_queue.cc:260] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7abb819105d41488715d1f3a2443227" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39837 } } peers { permanent_uuid: "84a5aca1a73948028d2a2b1bfc5bb0b8" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44147 } } peers { permanent_uuid: "0e33afc7b1634128b852099aa8eb6881" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46621 } }
I20260808 18:32:37.099521 17505 ts_tablet_manager.cc:1428] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 18:32:37.099529 17507 ts_tablet_manager.cc:1428] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 84a5aca1a73948028d2a2b1bfc5bb0b8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 18:32:37.099900 17486 heartbeater.cc:503] Master 127.29.78.190:39149 was elected leader, sending a full tablet report...
I20260808 18:32:37.100201 17199 heartbeater.cc:503] Master 127.29.78.190:39149 was elected leader, sending a full tablet report...
W20260808 18:32:37.155628 17201 tablet.cc:2369] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 84a5aca1a73948028d2a2b1bfc5bb0b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 18:32:37.280295 17487 tablet.cc:2369] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:32:37.327939 17511 raft_consensus.cc:482] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:32:37.328177 17511 raft_consensus.cc:504] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7abb819105d41488715d1f3a2443227" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39837 } } peers { permanent_uuid: "84a5aca1a73948028d2a2b1bfc5bb0b8" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44147 } } peers { permanent_uuid: "0e33afc7b1634128b852099aa8eb6881" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46621 } }
I20260808 18:32:37.329607 17511 leader_election.cc:288] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7abb819105d41488715d1f3a2443227 (127.29.78.131:39837), 84a5aca1a73948028d2a2b1bfc5bb0b8 (127.29.78.129:44147)
I20260808 18:32:37.331301 17513 raft_consensus.cc:482] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 84a5aca1a73948028d2a2b1bfc5bb0b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:32:37.331493 17513 raft_consensus.cc:504] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 84a5aca1a73948028d2a2b1bfc5bb0b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7abb819105d41488715d1f3a2443227" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39837 } } peers { permanent_uuid: "84a5aca1a73948028d2a2b1bfc5bb0b8" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44147 } } peers { permanent_uuid: "0e33afc7b1634128b852099aa8eb6881" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46621 } }
I20260808 18:32:37.332576 17513 leader_election.cc:288] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 84a5aca1a73948028d2a2b1bfc5bb0b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7abb819105d41488715d1f3a2443227 (127.29.78.131:39837), 0e33afc7b1634128b852099aa8eb6881 (127.29.78.130:46621)
I20260808 18:32:37.334611 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:37.329810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:39837 (local address 127.29.78.130:46307)
0808 18:32:37.330023 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:37.330039 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:37.330145 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:37.330490 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:37.330493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:37.330512 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:37.330655 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:37.330666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.331617 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.331623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:37.332895 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:37.332906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.334028 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.334032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:37.334239 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:37.334302 (+ 63us) client_negotiation.cc:769] Sending connection context
0808 18:32:37.334374 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 18:32:37.334476 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":57}
I20260808 18:32:37.334614 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:37.330077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:53193 (local address 127.29.78.129:44147)
0808 18:32:37.330219 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:37.330224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:37.330451 (+ 227us) server_negotiation.cc:407] Connection header received
0808 18:32:37.330568 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:37.330571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:37.330625 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:37.330710 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:37.331009 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.331680 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.333146 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.333786 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.333887 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:37.333945 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:37.334442 (+ 497us) server_negotiation.cc:299] Negotiation successful
0808 18:32:37.334521 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 18:32:37.335019 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:37.329947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:44147 (local address 127.29.78.130:53193)
0808 18:32:37.330331 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:37.330345 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:37.330477 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:37.330751 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:37.330754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:37.330772 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:37.330895 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:37.330901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.331794 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.331797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:37.332989 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:37.332999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.334028 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.334032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:37.334202 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:37.334298 (+ 96us) client_negotiation.cc:769] Sending connection context
0808 18:32:37.334782 (+ 484us) client_negotiation.cc:240] Negotiation successful
0808 18:32:37.334872 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":86,"thread_start_us":91,"threads_started":1}
I20260808 18:32:37.335433 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:37.329953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:46307 (local address 127.29.78.131:39837)
0808 18:32:37.330114 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:37.330119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:37.330138 (+ 19us) server_negotiation.cc:407] Connection header received
0808 18:32:37.330284 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:37.330288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:37.330344 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:37.330476 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:37.330769 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.331513 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.333160 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.333765 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.333887 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:37.333939 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:37.335246 (+ 1307us) server_negotiation.cc:299] Negotiation successful
0808 18:32:37.335317 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 18:32:37.335595 17151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7e307a2a1a40ccb3a1e0f91fbf73a8" candidate_uuid: "0e33afc7b1634128b852099aa8eb6881" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84a5aca1a73948028d2a2b1bfc5bb0b8" is_pre_election: true
I20260808 18:32:37.335816 17151 raft_consensus.cc:2454] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 84a5aca1a73948028d2a2b1bfc5bb0b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e33afc7b1634128b852099aa8eb6881 in term 0.
I20260808 18:32:37.335932 17437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7e307a2a1a40ccb3a1e0f91fbf73a8" candidate_uuid: "0e33afc7b1634128b852099aa8eb6881" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7abb819105d41488715d1f3a2443227" is_pre_election: true
I20260808 18:32:37.336213 17437 raft_consensus.cc:2454] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e33afc7b1634128b852099aa8eb6881 in term 0.
I20260808 18:32:37.336277 17222 leader_election.cc:302] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e33afc7b1634128b852099aa8eb6881, 84a5aca1a73948028d2a2b1bfc5bb0b8; no voters:
I20260808 18:32:37.336575 17511 raft_consensus.cc:2781] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:32:37.336653 17511 raft_consensus.cc:482] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:32:37.336694 17511 raft_consensus.cc:3037] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:37.337941 17511 raft_consensus.cc:504] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7abb819105d41488715d1f3a2443227" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39837 } } peers { permanent_uuid: "84a5aca1a73948028d2a2b1bfc5bb0b8" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44147 } } peers { permanent_uuid: "0e33afc7b1634128b852099aa8eb6881" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46621 } }
I20260808 18:32:37.338238 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:37.333633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:46963 (local address 127.29.78.130:46621)
0808 18:32:37.333778 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:37.333783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:37.333795 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:32:37.333835 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:37.333838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:37.333882 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:37.333959 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:37.334444 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.335206 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.336969 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.337559 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.337665 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:37.337705 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:37.338079 (+ 374us) server_negotiation.cc:299] Negotiation successful
0808 18:32:37.338143 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 18:32:37.338275 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:37.332845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:46621 (local address 127.29.78.129:46963)
0808 18:32:37.333396 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:37.333409 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:37.333517 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:37.334064 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:37.334068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:37.334087 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:37.334291 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:37.334298 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.335354 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.335357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:37.336824 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:37.336835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.337802 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.337805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:37.337914 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:37.337954 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 18:32:37.338042 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 18:32:37.338112 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":144,"spinlock_wait_cycles":17664,"thread_start_us":107,"threads_started":1}
I20260808 18:32:37.338639 17437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7e307a2a1a40ccb3a1e0f91fbf73a8" candidate_uuid: "0e33afc7b1634128b852099aa8eb6881" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7abb819105d41488715d1f3a2443227"
I20260808 18:32:37.338857 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:37.333040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:56471 (local address 127.29.78.131:39837)
0808 18:32:37.333778 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:37.333788 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:37.333801 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:32:37.333852 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:37.333854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:37.333895 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:37.333970 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:37.334423 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.335231 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.336969 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.337585 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.337691 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:37.337738 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:37.338164 (+ 426us) server_negotiation.cc:299] Negotiation successful
0808 18:32:37.338228 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":126,"threads_started":1}
I20260808 18:32:37.338862 17289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7e307a2a1a40ccb3a1e0f91fbf73a8" candidate_uuid: "84a5aca1a73948028d2a2b1bfc5bb0b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e33afc7b1634128b852099aa8eb6881" is_pre_election: true
I20260808 18:32:37.338766 17437 raft_consensus.cc:3037] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:37.339028 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:37.332701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:39837 (local address 127.29.78.129:56471)
0808 18:32:37.333314 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:37.333329 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:37.333430 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:37.334083 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:37.334086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:37.334098 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:37.334268 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:37.334277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.335354 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.335357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:37.336824 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:37.336835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.337739 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.337746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:37.337874 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:37.337949 (+ 75us) client_negotiation.cc:769] Sending connection context
0808 18:32:37.338041 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 18:32:37.338604 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":179,"thread_start_us":69,"threads_started":1}
I20260808 18:32:37.339138 17289 raft_consensus.cc:2379] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84a5aca1a73948028d2a2b1bfc5bb0b8 in current term 1: Already voted for candidate 0e33afc7b1634128b852099aa8eb6881 in this term.
W20260808 18:32:37.338925 17342 tablet.cc:2369] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:32:37.340076 17437 raft_consensus.cc:2454] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e33afc7b1634128b852099aa8eb6881 in term 1.
I20260808 18:32:37.340438 17221 leader_election.cc:302] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e33afc7b1634128b852099aa8eb6881, e7abb819105d41488715d1f3a2443227; no voters:
I20260808 18:32:37.340590 17437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7e307a2a1a40ccb3a1e0f91fbf73a8" candidate_uuid: "84a5aca1a73948028d2a2b1bfc5bb0b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7abb819105d41488715d1f3a2443227" is_pre_election: true
I20260808 18:32:37.340772 17437 raft_consensus.cc:2379] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84a5aca1a73948028d2a2b1bfc5bb0b8 in current term 1: Already voted for candidate 0e33afc7b1634128b852099aa8eb6881 in this term.
I20260808 18:32:37.340826 17518 raft_consensus.cc:2781] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:32:37.341030 17151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7e307a2a1a40ccb3a1e0f91fbf73a8" candidate_uuid: "0e33afc7b1634128b852099aa8eb6881" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84a5aca1a73948028d2a2b1bfc5bb0b8"
I20260808 18:32:37.341064 17518 raft_consensus.cc:686] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 [term 1 LEADER]: Becoming Leader. State: Replica: 0e33afc7b1634128b852099aa8eb6881, State: Running, Role: LEADER
I20260808 18:32:37.341158 17151 raft_consensus.cc:3037] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 84a5aca1a73948028d2a2b1bfc5bb0b8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:37.341271 17079 leader_election.cc:302] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 84a5aca1a73948028d2a2b1bfc5bb0b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84a5aca1a73948028d2a2b1bfc5bb0b8; no voters: 0e33afc7b1634128b852099aa8eb6881, e7abb819105d41488715d1f3a2443227
I20260808 18:32:37.341379 17518 consensus_queue.cc:237] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7abb819105d41488715d1f3a2443227" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39837 } } peers { permanent_uuid: "84a5aca1a73948028d2a2b1bfc5bb0b8" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44147 } } peers { permanent_uuid: "0e33afc7b1634128b852099aa8eb6881" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46621 } }
I20260808 18:32:37.342269 17151 raft_consensus.cc:2454] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 84a5aca1a73948028d2a2b1bfc5bb0b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e33afc7b1634128b852099aa8eb6881 in term 1.
I20260808 18:32:37.342381 17513 raft_consensus.cc:2726] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 84a5aca1a73948028d2a2b1bfc5bb0b8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 18:32:37.340917 17511 leader_election.cc:288] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 [CANDIDATE]: Term 1 election: Requested vote from peers e7abb819105d41488715d1f3a2443227 (127.29.78.131:39837), 84a5aca1a73948028d2a2b1bfc5bb0b8 (127.29.78.129:44147)
I20260808 18:32:37.344431 16992 catalog_manager.cc:5495] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e33afc7b1634128b852099aa8eb6881 (127.29.78.130). New cstate: current_term: 1 leader_uuid: "0e33afc7b1634128b852099aa8eb6881" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7abb819105d41488715d1f3a2443227" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84a5aca1a73948028d2a2b1bfc5bb0b8" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e33afc7b1634128b852099aa8eb6881" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46621 } health_report { overall_health: HEALTHY } } }
I20260808 18:32:37.368986 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:37.365801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43888 (local address 127.29.78.130:46621)
0808 18:32:37.365926 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:37.365930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:37.365942 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:32:37.366014 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:37.366016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:37.366072 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:37.366159 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:32:37.366363 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.366902 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.367716 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:37.367927 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:37.368001 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:37.368261 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:32:37.368673 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:32:37.368765 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:37.368819 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 18:32:37.368861 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 18:32:37.374769 17437 raft_consensus.cc:1261] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227 [term 1 FOLLOWER]: Refusing update from remote peer 0e33afc7b1634128b852099aa8eb6881: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:32:37.374816 17151 raft_consensus.cc:1261] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 84a5aca1a73948028d2a2b1bfc5bb0b8 [term 1 FOLLOWER]: Refusing update from remote peer 0e33afc7b1634128b852099aa8eb6881: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:32:37.375391 17511 consensus_queue.cc:1035] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7abb819105d41488715d1f3a2443227" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:32:37.375670 17518 consensus_queue.cc:1035] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84a5aca1a73948028d2a2b1bfc5bb0b8" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:32:37.383271 17524 mvcc.cc:205] Tried to move back new op lower bound from 7316332369400131584 to 7316332369272659968. Current Snapshot: MvccSnapshot[applied={T|T < 7316332369400131584}]
I20260808 18:32:37.385186 17523 mvcc.cc:205] Tried to move back new op lower bound from 7316332369400131584 to 7316332369272659968. Current Snapshot: MvccSnapshot[applied={T|T < 7316332369400131584}]
I20260808 18:32:37.391031 16995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:32:37.391211 16995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:32:37.393682 16995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f72bf1265ce40629e4c33681174e6e7: Sending DeleteTablet for 3 replicas of tablet ce7e307a2a1a40ccb3a1e0f91fbf73a8
I20260808 18:32:37.394505 17126 tablet_service.cc:1514] Processing DeleteTablet for tablet ce7e307a2a1a40ccb3a1e0f91fbf73a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:32:37 UTC) from {username='slave'} at 127.0.0.1:51536
I20260808 18:32:37.394802 17268 tablet_service.cc:1514] Processing DeleteTablet for tablet ce7e307a2a1a40ccb3a1e0f91fbf73a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:32:37 UTC) from {username='slave'} at 127.0.0.1:43886
I20260808 18:32:37.395049 17414 tablet_service.cc:1514] Processing DeleteTablet for tablet ce7e307a2a1a40ccb3a1e0f91fbf73a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:32:37 UTC) from {username='slave'} at 127.0.0.1:53914
I20260808 18:32:37.395426 17532 tablet_replica.cc:330] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881: stopping tablet replica
I20260808 18:32:37.395450 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 17059
I20260808 18:32:37.400413 17532 raft_consensus.cc:2229] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:32:37.400799 17532 raft_consensus.cc:2258] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:32:37.402153 17532 ts_tablet_manager.cc:1905] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:32:37.403759 17533 tablet_replica.cc:330] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227: stopping tablet replica
I20260808 18:32:37.404101 17533 raft_consensus.cc:2229] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:32:37.404429 17533 raft_consensus.cc:2258] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:32:37.405099 17532 ts_tablet_manager.cc:1918] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:32:37.405213 17532 log.cc:1196] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/ce7e307a2a1a40ccb3a1e0f91fbf73a8
I20260808 18:32:37.405516 17532 ts_tablet_manager.cc:1939] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P 0e33afc7b1634128b852099aa8eb6881: Deleting consensus metadata
I20260808 18:32:37.406649 16978 catalog_manager.cc:4841] TS 0e33afc7b1634128b852099aa8eb6881 (127.29.78.130:46621): tablet ce7e307a2a1a40ccb3a1e0f91fbf73a8 (table test-table [id=2f2e606e4ae4498ea5ede3582747733a]) successfully deleted
I20260808 18:32:37.407981 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 17204
W20260808 18:32:37.408000 16980 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:44147 (error 108)
I20260808 18:32:37.411669 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:37.411392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:44147 (local address 127.0.0.1:51548)
0808 18:32:37.411523 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:37.411595 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 18:32:37.411841 16980 catalog_manager.cc:4584] TS 84a5aca1a73948028d2a2b1bfc5bb0b8 (127.29.78.129:44147): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce7e307a2a1a40ccb3a1e0f91fbf73a8: Network error: Client connection negotiation failed: client connection to 127.29.78.129:44147: connect: Connection refused (error 111)
I20260808 18:32:37.417060 17533 ts_tablet_manager.cc:1905] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:32:37.420615 17533 ts_tablet_manager.cc:1918] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:32:37.420725 17533 log.cc:1196] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/ce7e307a2a1a40ccb3a1e0f91fbf73a8
I20260808 18:32:37.421082 17533 ts_tablet_manager.cc:1939] T ce7e307a2a1a40ccb3a1e0f91fbf73a8 P e7abb819105d41488715d1f3a2443227: Deleting consensus metadata
I20260808 18:32:37.422215 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 17344
I20260808 18:32:37.428957 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 16961
2026-08-08T18:32:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1226 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:32:37 dist-test-slave-gp6n krb5kdc[17537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:32:37 dist-test-slave-gp6n krb5kdc[17537](info): set up 2 sockets
Aug 08 18:32:37 dist-test-slave-gp6n krb5kdc[17537](info): commencing operation
krb5kdc: starting...
W20260808 18:32:38.534049 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:32:38 dist-test-slave-gp6n krb5kdc[17537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:32:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:32:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:32:38.663760 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:40235
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:40235
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:38.774714 17551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:38.774968 17551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:38.775007 17551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:38.778934 17551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:32:38.778985 17551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:38.779004 17551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:38.779021 17551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:32:38.779036 17551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:32:38.784256 17551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:40235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:40235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:38.785197 17551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:38.786151 17551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:38.794744 17557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:38.794705 17556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:38.794725 17559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:38.794962 17551 server_base.cc:1034] running on GCE node
I20260808 18:32:38.795643 17551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:38.796732 17551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:38.797922 17551 hybrid_clock.cc:648] HybridClock initialized: now 1786213958797906 us; error 30 us; skew 500 ppm
Aug 08 18:32:38 dist-test-slave-gp6n krb5kdc[17537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213958, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:38.800676 17551 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:32:38.801510 17551 webserver.cc:460] Webserver started at http://127.29.78.190:45051/ using document root <none> and password file <none>
I20260808 18:32:38.802062 17551 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:38.802152 17551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:38.802381 17551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:38.804415 17551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "f7cfb777711b4335908f4b20ddac226a"
format_stamp: "Formatted at 2026-08-08 18:32:38 on dist-test-slave-gp6n"
server_key: "af4a3c813b9664be38cdd2a6377607eb"
server_key_iv: "0d28a90319b3140ea0986f8ffaf80636"
server_key_version: "encryptionkey@0"
I20260808 18:32:38.805012 17551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "f7cfb777711b4335908f4b20ddac226a"
format_stamp: "Formatted at 2026-08-08 18:32:38 on dist-test-slave-gp6n"
server_key: "af4a3c813b9664be38cdd2a6377607eb"
server_key_iv: "0d28a90319b3140ea0986f8ffaf80636"
server_key_version: "encryptionkey@0"
I20260808 18:32:38.808760 17551 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 18:32:38.811235 17566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:38.812441 17551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:32:38.812594 17551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "f7cfb777711b4335908f4b20ddac226a"
format_stamp: "Formatted at 2026-08-08 18:32:38 on dist-test-slave-gp6n"
server_key: "af4a3c813b9664be38cdd2a6377607eb"
server_key_iv: "0d28a90319b3140ea0986f8ffaf80636"
server_key_version: "encryptionkey@0"
I20260808 18:32:38.812709 17551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:38.835566 17551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:38.838588 17551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:38.838768 17551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:38.845981 17551 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:40235
I20260808 18:32:38.846076 17624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:40235 every 8 connection(s)
I20260808 18:32:38.847085 17551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:32:38.849963 17625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:38.850574 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 17551
I20260808 18:32:38.850718 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:32:38.851064 30010 external_mini_cluster.cc:1428] Setting key 856016ab11bc4e9412e7f88c1d5c2dc1
I20260808 18:32:38.856365 17625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a: Bootstrap starting.
I20260808 18:32:38.859156 17625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:38.860149 17625 log.cc:824] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a: Log is configured to *not* fsync() on all Append() calls
Aug 08 18:32:38 dist-test-slave-gp6n krb5kdc[17537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:32:38.862335 17625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a: No bootstrap required, opened a new log
I20260808 18:32:38.865893 17628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:38.852602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53208 (local address 127.29.78.190:40235)
0808 18:32:38.853081 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:38.853097 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:38.853134 (+ 37us) server_negotiation.cc:407] Connection header received
0808 18:32:38.853802 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:38.853818 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:38.854150 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:38.854543 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:38.855387 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:38.856392 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:38.857656 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:38.858130 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:38.858249 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:38.861205 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:38.861210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:38.861224 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:38.861252 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:38.863048 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:38.863547 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:38.863552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:38.863557 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:38.863668 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:38.863955 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:38.863960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:38.863961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:38.864356 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:38.864585 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:38.864847 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 18:32:38.865146 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":142,"threads_started":1}
I20260808 18:32:38.865978 17625 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cfb777711b4335908f4b20ddac226a" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40235 } }
I20260808 18:32:38.866289 17625 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:38.866369 17625 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7cfb777711b4335908f4b20ddac226a, State: Initialized, Role: FOLLOWER
I20260808 18:32:38.866904 17625 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cfb777711b4335908f4b20ddac226a" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40235 } }
I20260808 18:32:38.867075 17625 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:32:38.867148 17625 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:32:38.867246 17625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:38.868428 17625 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cfb777711b4335908f4b20ddac226a" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40235 } }
I20260808 18:32:38.868749 17625 leader_election.cc:302] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7cfb777711b4335908f4b20ddac226a; no voters:
I20260808 18:32:38.869020 17625 leader_election.cc:288] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:32:38.869146 17631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:32:38.869400 17631 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a [term 1 LEADER]: Becoming Leader. State: Replica: f7cfb777711b4335908f4b20ddac226a, State: Running, Role: LEADER
I20260808 18:32:38.869956 17631 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cfb777711b4335908f4b20ddac226a" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40235 } }
I20260808 18:32:38.870309 17625 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:32:38.871656 17632 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7cfb777711b4335908f4b20ddac226a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cfb777711b4335908f4b20ddac226a" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40235 } } }
I20260808 18:32:38.871799 17632 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:38.872108 17633 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7cfb777711b4335908f4b20ddac226a. Latest consensus state: current_term: 1 leader_uuid: "f7cfb777711b4335908f4b20ddac226a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cfb777711b4335908f4b20ddac226a" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40235 } } }
I20260808 18:32:38.872226 17633 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a [sys.catalog]: This master's current role is: LEADER
W20260808 18:32:38.875047 17648 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:32:38.875139 17648 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:32:38.875211 17643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:32:38.875908 17643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:32:38.881361 17643 catalog_manager.cc:1347] Generated new cluster ID: d9ea11c9ba8b465cb1d477ca2ec1c1d2
I20260808 18:32:38.881420 17643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:32:38.886986 17643 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:32:38.888110 17643 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:32:38.893155 17643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a: Generated new TSK 0
I20260808 18:32:38.893748 17643 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260808 18:32:38.927904 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40235
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:39.041500 17654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:39.041745 17654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:39.041785 17654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:39.045828 17654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:39.045883 17654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:39.045976 17654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:32:39.051111 17654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:39.052101 17654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:39.053107 17654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:39.062973 17661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:39.062968 17660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:39.062973 17663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:39.063079 17654 server_base.cc:1034] running on GCE node
I20260808 18:32:39.063584 17654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:39.064296 17654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:39.065461 17654 hybrid_clock.cc:648] HybridClock initialized: now 1786213959065435 us; error 37 us; skew 500 ppm
Aug 08 18:32:39 dist-test-slave-gp6n krb5kdc[17537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213959, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:39.068286 17654 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:32:39.069171 17654 webserver.cc:460] Webserver started at http://127.29.78.129:35127/ using document root <none> and password file <none>
I20260808 18:32:39.069756 17654 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:39.069816 17654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:39.070029 17654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:39.072242 17654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "214b1244378d49d6b3e069ad254cad68"
format_stamp: "Formatted at 2026-08-08 18:32:39 on dist-test-slave-gp6n"
server_key: "35002bb173287adf7ccb025eb3a1e410"
server_key_iv: "dbf6362efcc5af634a7fdfa44636ed8a"
server_key_version: "encryptionkey@0"
I20260808 18:32:39.072836 17654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "214b1244378d49d6b3e069ad254cad68"
format_stamp: "Formatted at 2026-08-08 18:32:39 on dist-test-slave-gp6n"
server_key: "35002bb173287adf7ccb025eb3a1e410"
server_key_iv: "dbf6362efcc5af634a7fdfa44636ed8a"
server_key_version: "encryptionkey@0"
I20260808 18:32:39.076566 17654 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260808 18:32:39.078995 17671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:39.080194 17654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:32:39.080327 17654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "214b1244378d49d6b3e069ad254cad68"
format_stamp: "Formatted at 2026-08-08 18:32:39 on dist-test-slave-gp6n"
server_key: "35002bb173287adf7ccb025eb3a1e410"
server_key_iv: "dbf6362efcc5af634a7fdfa44636ed8a"
server_key_version: "encryptionkey@0"
I20260808 18:32:39.080430 17654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:39.107174 17654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:39.110237 17654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:39.110469 17654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:39.111100 17654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:32:39.111974 17654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:32:39.112025 17654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:39.112093 17654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:32:39.112123 17654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:39.121192 17654 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:38699
I20260808 18:32:39.121254 17798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:38699 every 8 connection(s)
I20260808 18:32:39.122139 17654 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:32:39.124933 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 17654
I20260808 18:32:39.125099 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:32:39.125463 30010 external_mini_cluster.cc:1428] Setting key 1f2a019b590250f556e12874998bce3a
Aug 08 18:32:39 dist-test-slave-gp6n krb5kdc[17537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213959, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260808 18:32:39.137526 17628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:39.123960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:52625 (local address 127.29.78.190:40235)
0808 18:32:39.124125 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:39.124130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:39.125141 (+ 1011us) server_negotiation.cc:407] Connection header received
0808 18:32:39.126781 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:39.126784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:39.126850 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:39.126934 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:39.128253 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.129108 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.130322 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.130622 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.130742 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:39.133711 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:39.133714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:39.133716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:39.133740 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:39.135297 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:39.135908 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:39.135911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:39.135912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:39.135985 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:39.136389 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:39.136391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:39.136392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:39.136560 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:39.136636 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:39.137286 (+ 650us) server_negotiation.cc:299] Negotiation successful
0808 18:32:39.137399 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 18:32:39.138360 17802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:39.124395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40235 (local address 127.29.78.129:52625)
0808 18:32:39.124956 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:39.125022 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:39.126537 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:39.127188 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:39.127198 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:39.127587 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:32:39.128081 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:39.128095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.129271 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.129274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:39.130187 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:39.130197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.130783 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.130788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:39.130964 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:39.131601 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:32:39.131617 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:32:39.133536 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:32:39.135423 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:32:39.135432 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:32:39.135449 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:32:39.135805 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:32:39.136085 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:32:39.136088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:32:39.136090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:32:39.136289 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:32:39.136669 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:32:39.136676 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:32:39.137020 (+ 344us) client_negotiation.cc:769] Sending connection context
0808 18:32:39.137239 (+ 219us) client_negotiation.cc:240] Negotiation successful
0808 18:32:39.137501 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":128,"threads_started":1}
I20260808 18:32:39.139710 17799 heartbeater.cc:347] Connected to a master server at 127.29.78.190:40235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
I20260808 18:32:39.139964 17799 heartbeater.cc:464] Registering TS with master...
I20260808 18:32:39.140465 17799 heartbeater.cc:511] Master 127.29.78.190:40235 requested a full tablet report, sending...
I20260808 18:32:39.142199 17583 ts_manager.cc:194] Registered new tserver with Master: 214b1244378d49d6b3e069ad254cad68 (127.29.78.129:38699)
I20260808 18:32:39.143461 17583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:52625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260808 18:32:39.157866 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40235
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:39.269272 17805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:39.269528 17805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:39.269570 17805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:39.273779 17805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:39.273831 17805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:39.273926 17805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:32:39.279035 17805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:39.280010 17805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:39.280993 17805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:39.290639 17810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:39.290652 17813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:39.290730 17805 server_base.cc:1034] running on GCE node
W20260808 18:32:39.290663 17811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:39.291324 17805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:39.291978 17805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:39.293181 17805 hybrid_clock.cc:648] HybridClock initialized: now 1786213959293152 us; error 41 us; skew 500 ppm
Aug 08 18:32:39 dist-test-slave-gp6n krb5kdc[17537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213959, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:39.296010 17805 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:32:39.296857 17805 webserver.cc:460] Webserver started at http://127.29.78.130:33389/ using document root <none> and password file <none>
I20260808 18:32:39.297427 17805 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:39.297518 17805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:39.297745 17805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:39.299834 17805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "140fc3d6b425431397bb971191bf6522"
format_stamp: "Formatted at 2026-08-08 18:32:39 on dist-test-slave-gp6n"
server_key: "bb120d5bac44bdcb17012fd6089bc8ad"
server_key_iv: "185dc3413c13fc82764c59a3b55c8141"
server_key_version: "encryptionkey@0"
I20260808 18:32:39.300411 17805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "140fc3d6b425431397bb971191bf6522"
format_stamp: "Formatted at 2026-08-08 18:32:39 on dist-test-slave-gp6n"
server_key: "bb120d5bac44bdcb17012fd6089bc8ad"
server_key_iv: "185dc3413c13fc82764c59a3b55c8141"
server_key_version: "encryptionkey@0"
I20260808 18:32:39.304124 17805 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 18:32:39.306448 17822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:39.307747 17805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:32:39.307840 17805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "140fc3d6b425431397bb971191bf6522"
format_stamp: "Formatted at 2026-08-08 18:32:39 on dist-test-slave-gp6n"
server_key: "bb120d5bac44bdcb17012fd6089bc8ad"
server_key_iv: "185dc3413c13fc82764c59a3b55c8141"
server_key_version: "encryptionkey@0"
I20260808 18:32:39.307919 17805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:39.336154 17805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:39.339237 17805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:39.339398 17805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:39.339954 17805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:32:39.340758 17805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:32:39.340854 17805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:39.340909 17805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:32:39.340936 17805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:39.349792 17805 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:45277
I20260808 18:32:39.349820 17943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:45277 every 8 connection(s)
I20260808 18:32:39.350783 17805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:32:39.354718 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 17805
I20260808 18:32:39.354828 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:32:39.355145 30010 external_mini_cluster.cc:1428] Setting key 91382771866e97e13d2b05fc22b1e287
Aug 08 18:32:39 dist-test-slave-gp6n krb5kdc[17537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213959, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:32:39.365845 17628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:39.352667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:48247 (local address 127.29.78.190:40235)
0808 18:32:39.352812 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:39.352816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:39.353657 (+ 841us) server_negotiation.cc:407] Connection header received
0808 18:32:39.355081 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:39.355084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:39.355145 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:39.355230 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:39.356785 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.357564 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.358759 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.359147 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.359232 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:39.362217 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:39.362221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:39.362224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:39.362256 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:39.363774 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:39.364295 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:39.364300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:39.364301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:39.364385 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:39.364741 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:39.364746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:39.364747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:39.364932 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:39.365051 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:39.365543 (+ 492us) server_negotiation.cc:299] Negotiation successful
0808 18:32:39.365669 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260808 18:32:39.366760 17946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:39.353037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40235 (local address 127.29.78.130:48247)
0808 18:32:39.353488 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:39.353533 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:39.354798 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:39.355414 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:39.355422 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:39.355914 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:32:39.356616 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:39.356639 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.357696 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.357698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:39.358628 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:39.358637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.359313 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.359319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:39.359548 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:39.360200 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:32:39.360226 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:32:39.362047 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:32:39.363895 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:32:39.363901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:32:39.363918 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:32:39.364188 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:32:39.364489 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:32:39.364492 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:32:39.364494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:32:39.364643 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:32:39.365043 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:32:39.365050 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:32:39.365300 (+ 250us) client_negotiation.cc:769] Sending connection context
0808 18:32:39.365489 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 18:32:39.365820 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":120,"threads_started":1}
I20260808 18:32:39.368106 17944 heartbeater.cc:347] Connected to a master server at 127.29.78.190:40235
I20260808 18:32:39.368371 17944 heartbeater.cc:464] Registering TS with master...
I20260808 18:32:39.368882 17944 heartbeater.cc:511] Master 127.29.78.190:40235 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
I20260808 18:32:39.370021 17583 ts_manager.cc:194] Registered new tserver with Master: 140fc3d6b425431397bb971191bf6522 (127.29.78.130:45277)
I20260808 18:32:39.370740 17583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:48247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260808 18:32:39.385670 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40235
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:39.495435 17949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:39.495678 17949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:39.495720 17949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:39.499626 17949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:39.499676 17949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:39.499730 17949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:32:39.504809 17949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:39.505702 17949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:39.506723 17949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:39.515993 17956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:39.516009 17959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:39.516081 17955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:39.516368 17949 server_base.cc:1034] running on GCE node
I20260808 18:32:39.516796 17949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:39.517477 17949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:39.518688 17949 hybrid_clock.cc:648] HybridClock initialized: now 1786213959518670 us; error 30 us; skew 500 ppm
Aug 08 18:32:39 dist-test-slave-gp6n krb5kdc[17537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213959, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:39.521356 17949 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:32:39.522249 17949 webserver.cc:460] Webserver started at http://127.29.78.131:34401/ using document root <none> and password file <none>
I20260808 18:32:39.522853 17949 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:39.522948 17949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:39.523170 17949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:39.525085 17949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "5a28acc0c9294f36a7c9a01ad7aee8aa"
format_stamp: "Formatted at 2026-08-08 18:32:39 on dist-test-slave-gp6n"
server_key: "59202c16451bae939a1bebb707c0678a"
server_key_iv: "d5857a51bc9b531f2cb74400d6f9f641"
server_key_version: "encryptionkey@0"
I20260808 18:32:39.525667 17949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "5a28acc0c9294f36a7c9a01ad7aee8aa"
format_stamp: "Formatted at 2026-08-08 18:32:39 on dist-test-slave-gp6n"
server_key: "59202c16451bae939a1bebb707c0678a"
server_key_iv: "d5857a51bc9b531f2cb74400d6f9f641"
server_key_version: "encryptionkey@0"
I20260808 18:32:39.529354 17949 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 18:32:39.531816 17966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:39.533056 17949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:32:39.533188 17949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "5a28acc0c9294f36a7c9a01ad7aee8aa"
format_stamp: "Formatted at 2026-08-08 18:32:39 on dist-test-slave-gp6n"
server_key: "59202c16451bae939a1bebb707c0678a"
server_key_iv: "d5857a51bc9b531f2cb74400d6f9f641"
server_key_version: "encryptionkey@0"
I20260808 18:32:39.533303 17949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:39.561543 17949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:39.564637 17949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:39.564844 17949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:39.565438 17949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:32:39.566259 17949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:32:39.566331 17949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:39.566423 17949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:32:39.566468 17949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:39.576130 17949 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:45507
I20260808 18:32:39.576159 18086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:45507 every 8 connection(s)
I20260808 18:32:39.577042 17949 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:32:39.582706 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 17949
I20260808 18:32:39.582857 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:32:39.583165 30010 external_mini_cluster.cc:1428] Setting key 730a063c6f3184b9b031c19d2dea4da0
Aug 08 18:32:39 dist-test-slave-gp6n krb5kdc[17537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213959, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:32:39.590852 17628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:39.578876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:38089 (local address 127.29.78.190:40235)
0808 18:32:39.579047 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:39.579051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:39.579894 (+ 843us) server_negotiation.cc:407] Connection header received
0808 18:32:39.581193 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:39.581196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:39.581255 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:39.581336 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:39.582685 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.583400 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.584938 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.585197 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.585309 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:39.587823 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:39.587826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:39.587828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:39.587852 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:39.589061 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:39.589519 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:39.589521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:39.589523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:39.589599 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:39.589888 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:39.589891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:39.589892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:39.590059 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:39.590133 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:39.590635 (+ 502us) server_negotiation.cc:299] Negotiation successful
0808 18:32:39.590732 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 18:32:39.591750 18091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:39.579281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40235 (local address 127.29.78.131:38089)
0808 18:32:39.579729 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:39.579771 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:39.580984 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:39.581511 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:39.581517 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:39.581863 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:32:39.582540 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:39.582557 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.583525 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.583528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:39.584806 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:39.584817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.585379 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.585384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:39.585553 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:39.586166 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:32:39.586183 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:32:39.587673 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:32:39.589160 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:32:39.589165 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:32:39.589176 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:32:39.589429 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:32:39.589676 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:32:39.589678 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:32:39.589680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:32:39.589811 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:32:39.590148 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:32:39.590152 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:32:39.590365 (+ 213us) client_negotiation.cc:769] Sending connection context
0808 18:32:39.590588 (+ 223us) client_negotiation.cc:240] Negotiation successful
0808 18:32:39.590832 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":116,"threads_started":1}
I20260808 18:32:39.592837 18087 heartbeater.cc:347] Connected to a master server at 127.29.78.190:40235
I20260808 18:32:39.593071 18087 heartbeater.cc:464] Registering TS with master...
I20260808 18:32:39.593545 18087 heartbeater.cc:511] Master 127.29.78.190:40235 requested a full tablet report, sending...
I20260808 18:32:39.594542 17583 ts_manager.cc:194] Registered new tserver with Master: 5a28acc0c9294f36a7c9a01ad7aee8aa (127.29.78.131:45507)
I20260808 18:32:39.595093 17583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:38089
I20260808 18:32:39.598096 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:32:39.607237 17628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:39.600074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53214 (local address 127.29.78.190:40235)
0808 18:32:39.600186 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:39.600188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:39.600408 (+ 220us) server_negotiation.cc:407] Connection header received
0808 18:32:39.600494 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:39.600496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:39.600530 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:39.600593 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:39.601233 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.601725 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.602598 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.602799 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.602875 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:39.604296 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:39.604298 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:39.604300 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:39.604320 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:39.605737 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:39.606172 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:39.606175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:39.606176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:39.606244 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:39.606566 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:39.606568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:39.606569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:39.606749 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:39.606864 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:39.607063 (+ 199us) server_negotiation.cc:299] Negotiation successful
0808 18:32:39.607134 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:32:39.609838 17583 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:32:39.612937 17583 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:32:39.626816 18103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:39.620723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:45277 (local address 127.0.0.1:53274)
0808 18:32:39.621168 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:39.621181 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:39.621254 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:39.622627 (+ 1373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:39.622629 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:39.622644 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:39.622765 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:39.622772 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.623862 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.623867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:39.625412 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:39.625422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.626234 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.626238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:39.626355 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:39.626396 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 18:32:39.626529 (+ 133us) client_negotiation.cc:240] Negotiation successful
0808 18:32:39.626654 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":211,"thread_start_us":93,"threads_started":1}
I20260808 18:32:39.626937 18101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:39.620522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:38699 (local address 127.0.0.1:33696)
0808 18:32:39.621015 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:39.621063 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:39.621190 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:39.622461 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:39.622466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:39.622498 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:39.622646 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:39.622654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.624147 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.624150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:39.625347 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:39.625358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.626472 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.626478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:39.626613 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:39.626635 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 18:32:39.626702 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 18:32:39.626761 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":23,"thread_start_us":65,"threads_started":1}
I20260808 18:32:39.626945 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:39.621570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53274 (local address 127.29.78.130:45277)
0808 18:32:39.622131 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:39.622144 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:39.622169 (+ 25us) server_negotiation.cc:407] Connection header received
0808 18:32:39.622248 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:39.622252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:39.622479 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:39.622640 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:39.622869 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.623732 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.625550 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.626102 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.626223 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:39.626378 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:39.626704 (+ 326us) server_negotiation.cc:299] Negotiation successful
0808 18:32:39.626804 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":142,"threads_started":1}
I20260808 18:32:39.627341 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:39.621359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36256 (local address 127.29.78.131:45507)
0808 18:32:39.621945 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:39.621957 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:39.621975 (+ 18us) server_negotiation.cc:407] Connection header received
0808 18:32:39.622045 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:39.622051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:39.622250 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:39.622431 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:39.622995 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.624093 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.625436 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.626289 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.626901 (+ 612us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:39.627053 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:39.627109 (+ 56us) server_negotiation.cc:299] Negotiation successful
0808 18:32:39.627202 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":106,"threads_started":1}
I20260808 18:32:39.627596 18102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:39.620679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33696 (local address 127.29.78.129:38699)
0808 18:32:39.621929 (+ 1250us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:39.621945 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:39.621971 (+ 26us) server_negotiation.cc:407] Connection header received
0808 18:32:39.622045 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:39.622051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:39.622250 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:39.622431 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:39.622784 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.624009 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.625482 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.626288 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.626474 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:39.626615 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:39.627268 (+ 653us) server_negotiation.cc:299] Negotiation successful
0808 18:32:39.627369 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1131,"thread_start_us":110,"threads_started":1}
I20260808 18:32:39.630297 17873 tablet_service.cc:1467] Processing CreateTablet for tablet 6ffa1bfe6ca644cb819783776611a52a (DEFAULT_TABLE table=test-table [id=a6810993010341ed8e37a7259c56ebce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:32:39.630357 17725 tablet_service.cc:1467] Processing CreateTablet for tablet 6ffa1bfe6ca644cb819783776611a52a (DEFAULT_TABLE table=test-table [id=a6810993010341ed8e37a7259c56ebce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:32:39.631491 17873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ffa1bfe6ca644cb819783776611a52a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:39.631511 17725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ffa1bfe6ca644cb819783776611a52a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:39.634584 18104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:39.621205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:45507 (local address 127.0.0.1:36256)
0808 18:32:39.621618 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:39.621632 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:39.621712 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:39.622757 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:39.622759 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:39.622780 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:39.622872 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:39.622878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.624211 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.624214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:39.625294 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:39.625303 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.626464 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.626468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:39.626561 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:39.626607 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 18:32:39.626660 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 18:32:39.626734 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":6,"thread_start_us":101,"threads_started":1}
I20260808 18:32:39.637238 18018 tablet_service.cc:1467] Processing CreateTablet for tablet 6ffa1bfe6ca644cb819783776611a52a (DEFAULT_TABLE table=test-table [id=a6810993010341ed8e37a7259c56ebce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:32:39.638290 18018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ffa1bfe6ca644cb819783776611a52a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:39.639269 18108 tablet_bootstrap.cc:492] T 6ffa1bfe6ca644cb819783776611a52a P 214b1244378d49d6b3e069ad254cad68: Bootstrap starting.
I20260808 18:32:39.639636 18107 tablet_bootstrap.cc:492] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522: Bootstrap starting.
I20260808 18:32:39.642071 18108 tablet_bootstrap.cc:654] T 6ffa1bfe6ca644cb819783776611a52a P 214b1244378d49d6b3e069ad254cad68: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:39.643047 18108 log.cc:824] T 6ffa1bfe6ca644cb819783776611a52a P 214b1244378d49d6b3e069ad254cad68: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:39.644513 18107 tablet_bootstrap.cc:654] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:39.645186 18108 tablet_bootstrap.cc:492] T 6ffa1bfe6ca644cb819783776611a52a P 214b1244378d49d6b3e069ad254cad68: No bootstrap required, opened a new log
I20260808 18:32:39.645392 18108 ts_tablet_manager.cc:1397] T 6ffa1bfe6ca644cb819783776611a52a P 214b1244378d49d6b3e069ad254cad68: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:32:39.645666 18107 log.cc:824] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:39.647399 18107 tablet_bootstrap.cc:492] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522: No bootstrap required, opened a new log
I20260808 18:32:39.647594 18107 ts_tablet_manager.cc:1397] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260808 18:32:39.649026 18108 raft_consensus.cc:348] T 6ffa1bfe6ca644cb819783776611a52a P 214b1244378d49d6b3e069ad254cad68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214b1244378d49d6b3e069ad254cad68" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38699 } } peers { permanent_uuid: "140fc3d6b425431397bb971191bf6522" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 45277 } } peers { permanent_uuid: "5a28acc0c9294f36a7c9a01ad7aee8aa" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45507 } }
I20260808 18:32:39.649274 18110 tablet_bootstrap.cc:492] T 6ffa1bfe6ca644cb819783776611a52a P 5a28acc0c9294f36a7c9a01ad7aee8aa: Bootstrap starting.
I20260808 18:32:39.649349 18108 raft_consensus.cc:374] T 6ffa1bfe6ca644cb819783776611a52a P 214b1244378d49d6b3e069ad254cad68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:39.649410 18108 raft_consensus.cc:729] T 6ffa1bfe6ca644cb819783776611a52a P 214b1244378d49d6b3e069ad254cad68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214b1244378d49d6b3e069ad254cad68, State: Initialized, Role: FOLLOWER
I20260808 18:32:39.649890 18108 consensus_queue.cc:260] T 6ffa1bfe6ca644cb819783776611a52a P 214b1244378d49d6b3e069ad254cad68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214b1244378d49d6b3e069ad254cad68" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38699 } } peers { permanent_uuid: "140fc3d6b425431397bb971191bf6522" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 45277 } } peers { permanent_uuid: "5a28acc0c9294f36a7c9a01ad7aee8aa" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45507 } }
I20260808 18:32:39.650611 17799 heartbeater.cc:503] Master 127.29.78.190:40235 was elected leader, sending a full tablet report...
I20260808 18:32:39.650935 18108 ts_tablet_manager.cc:1428] T 6ffa1bfe6ca644cb819783776611a52a P 214b1244378d49d6b3e069ad254cad68: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260808 18:32:39.651017 18107 raft_consensus.cc:348] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214b1244378d49d6b3e069ad254cad68" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38699 } } peers { permanent_uuid: "140fc3d6b425431397bb971191bf6522" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 45277 } } peers { permanent_uuid: "5a28acc0c9294f36a7c9a01ad7aee8aa" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45507 } }
I20260808 18:32:39.651299 18107 raft_consensus.cc:374] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:39.651348 18107 raft_consensus.cc:729] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140fc3d6b425431397bb971191bf6522, State: Initialized, Role: FOLLOWER
I20260808 18:32:39.651480 18110 tablet_bootstrap.cc:654] T 6ffa1bfe6ca644cb819783776611a52a P 5a28acc0c9294f36a7c9a01ad7aee8aa: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:39.651788 18107 consensus_queue.cc:260] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214b1244378d49d6b3e069ad254cad68" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38699 } } peers { permanent_uuid: "140fc3d6b425431397bb971191bf6522" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 45277 } } peers { permanent_uuid: "5a28acc0c9294f36a7c9a01ad7aee8aa" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45507 } }
I20260808 18:32:39.652467 17944 heartbeater.cc:503] Master 127.29.78.190:40235 was elected leader, sending a full tablet report...
I20260808 18:32:39.652726 18107 ts_tablet_manager.cc:1428] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260808 18:32:39.652721 18110 log.cc:824] T 6ffa1bfe6ca644cb819783776611a52a P 5a28acc0c9294f36a7c9a01ad7aee8aa: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:39.654850 18110 tablet_bootstrap.cc:492] T 6ffa1bfe6ca644cb819783776611a52a P 5a28acc0c9294f36a7c9a01ad7aee8aa: No bootstrap required, opened a new log
I20260808 18:32:39.655045 18110 ts_tablet_manager.cc:1397] T 6ffa1bfe6ca644cb819783776611a52a P 5a28acc0c9294f36a7c9a01ad7aee8aa: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260808 18:32:39.655455 18113 raft_consensus.cc:482] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:32:39.655601 18113 raft_consensus.cc:504] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214b1244378d49d6b3e069ad254cad68" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38699 } } peers { permanent_uuid: "140fc3d6b425431397bb971191bf6522" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 45277 } } peers { permanent_uuid: "5a28acc0c9294f36a7c9a01ad7aee8aa" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45507 } }
I20260808 18:32:39.656586 18113 leader_election.cc:288] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 214b1244378d49d6b3e069ad254cad68 (127.29.78.129:38699), 5a28acc0c9294f36a7c9a01ad7aee8aa (127.29.78.131:45507)
I20260808 18:32:39.658509 18110 raft_consensus.cc:348] T 6ffa1bfe6ca644cb819783776611a52a P 5a28acc0c9294f36a7c9a01ad7aee8aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214b1244378d49d6b3e069ad254cad68" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38699 } } peers { permanent_uuid: "140fc3d6b425431397bb971191bf6522" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 45277 } } peers { permanent_uuid: "5a28acc0c9294f36a7c9a01ad7aee8aa" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45507 } }
I20260808 18:32:39.658766 18110 raft_consensus.cc:374] T 6ffa1bfe6ca644cb819783776611a52a P 5a28acc0c9294f36a7c9a01ad7aee8aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:39.658839 18110 raft_consensus.cc:729] T 6ffa1bfe6ca644cb819783776611a52a P 5a28acc0c9294f36a7c9a01ad7aee8aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a28acc0c9294f36a7c9a01ad7aee8aa, State: Initialized, Role: FOLLOWER
I20260808 18:32:39.659257 18110 consensus_queue.cc:260] T 6ffa1bfe6ca644cb819783776611a52a P 5a28acc0c9294f36a7c9a01ad7aee8aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214b1244378d49d6b3e069ad254cad68" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38699 } } peers { permanent_uuid: "140fc3d6b425431397bb971191bf6522" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 45277 } } peers { permanent_uuid: "5a28acc0c9294f36a7c9a01ad7aee8aa" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45507 } }
I20260808 18:32:39.659871 18087 heartbeater.cc:503] Master 127.29.78.190:40235 was elected leader, sending a full tablet report...
I20260808 18:32:39.660203 18110 ts_tablet_manager.cc:1428] T 6ffa1bfe6ca644cb819783776611a52a P 5a28acc0c9294f36a7c9a01ad7aee8aa: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 18:32:39.661051 18102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:39.656977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:38733 (local address 127.29.78.129:38699)
0808 18:32:39.657085 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:39.657089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:39.657102 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:32:39.657190 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:39.657193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:39.657238 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:39.657317 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:39.657743 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.658494 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.659923 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.660506 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.660629 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:39.660679 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:39.660873 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 18:32:39.660940 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 18:32:39.661051 17946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:39.656729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:38699 (local address 127.29.78.130:38733)
0808 18:32:39.657004 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:39.657021 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:39.657103 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:39.657336 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:39.657339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:39.657356 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:39.657475 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:39.657480 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.658616 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.658619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:39.659783 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:39.659793 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.660629 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.660633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:39.660744 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:39.660773 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 18:32:39.660858 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 18:32:39.660937 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260808 18:32:39.661846 17748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ffa1bfe6ca644cb819783776611a52a" candidate_uuid: "140fc3d6b425431397bb971191bf6522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214b1244378d49d6b3e069ad254cad68" is_pre_election: true
I20260808 18:32:39.662156 17748 raft_consensus.cc:2454] T 6ffa1bfe6ca644cb819783776611a52a P 214b1244378d49d6b3e069ad254cad68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 140fc3d6b425431397bb971191bf6522 in term 0.
I20260808 18:32:39.662474 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:39.657164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:34779 (local address 127.29.78.131:45507)
0808 18:32:39.657267 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:39.657271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:39.658667 (+ 1396us) server_negotiation.cc:407] Connection header received
0808 18:32:39.658860 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:39.658864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:39.658931 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:39.658980 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:39.659379 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.660091 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.661385 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.661908 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.662041 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:39.662098 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:39.662287 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 18:32:39.662363 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 18:32:39.662488 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:39.656828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:45507 (local address 127.29.78.130:34779)
0808 18:32:39.658585 (+ 1757us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:39.658599 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:39.658734 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:39.659127 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:39.659130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:39.659147 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:39.659262 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:39.659269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.660323 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.660326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:39.661258 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:39.661267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.662051 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.662055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:39.662158 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:39.662182 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 18:32:39.662274 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 18:32:39.662339 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1681,"thread_start_us":102,"threads_started":1}
I20260808 18:32:39.662675 17824 leader_election.cc:302] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 140fc3d6b425431397bb971191bf6522, 214b1244378d49d6b3e069ad254cad68; no voters:
I20260808 18:32:39.663036 18038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ffa1bfe6ca644cb819783776611a52a" candidate_uuid: "140fc3d6b425431397bb971191bf6522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a28acc0c9294f36a7c9a01ad7aee8aa" is_pre_election: true
I20260808 18:32:39.663215 18038 raft_consensus.cc:2454] T 6ffa1bfe6ca644cb819783776611a52a P 5a28acc0c9294f36a7c9a01ad7aee8aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 140fc3d6b425431397bb971191bf6522 in term 0.
I20260808 18:32:39.663760 18113 raft_consensus.cc:2781] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:32:39.663872 18113 raft_consensus.cc:482] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:32:39.663941 18113 raft_consensus.cc:3037] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:39.665087 18113 raft_consensus.cc:504] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214b1244378d49d6b3e069ad254cad68" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38699 } } peers { permanent_uuid: "140fc3d6b425431397bb971191bf6522" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 45277 } } peers { permanent_uuid: "5a28acc0c9294f36a7c9a01ad7aee8aa" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45507 } }
I20260808 18:32:39.665457 18113 leader_election.cc:288] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522 [CANDIDATE]: Term 1 election: Requested vote from peers 214b1244378d49d6b3e069ad254cad68 (127.29.78.129:38699), 5a28acc0c9294f36a7c9a01ad7aee8aa (127.29.78.131:45507)
I20260808 18:32:39.665766 17748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ffa1bfe6ca644cb819783776611a52a" candidate_uuid: "140fc3d6b425431397bb971191bf6522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214b1244378d49d6b3e069ad254cad68"
I20260808 18:32:39.665917 17748 raft_consensus.cc:3037] T 6ffa1bfe6ca644cb819783776611a52a P 214b1244378d49d6b3e069ad254cad68 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:39.665971 18038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ffa1bfe6ca644cb819783776611a52a" candidate_uuid: "140fc3d6b425431397bb971191bf6522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a28acc0c9294f36a7c9a01ad7aee8aa"
I20260808 18:32:39.666111 18038 raft_consensus.cc:3037] T 6ffa1bfe6ca644cb819783776611a52a P 5a28acc0c9294f36a7c9a01ad7aee8aa [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:39.667039 17748 raft_consensus.cc:2454] T 6ffa1bfe6ca644cb819783776611a52a P 214b1244378d49d6b3e069ad254cad68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 140fc3d6b425431397bb971191bf6522 in term 1.
I20260808 18:32:39.667201 18038 raft_consensus.cc:2454] T 6ffa1bfe6ca644cb819783776611a52a P 5a28acc0c9294f36a7c9a01ad7aee8aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 140fc3d6b425431397bb971191bf6522 in term 1.
I20260808 18:32:39.667336 17824 leader_election.cc:302] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 140fc3d6b425431397bb971191bf6522, 214b1244378d49d6b3e069ad254cad68; no voters:
I20260808 18:32:39.667505 18113 raft_consensus.cc:2781] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:32:39.667765 18113 raft_consensus.cc:686] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522 [term 1 LEADER]: Becoming Leader. State: Replica: 140fc3d6b425431397bb971191bf6522, State: Running, Role: LEADER
I20260808 18:32:39.668105 18113 consensus_queue.cc:237] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214b1244378d49d6b3e069ad254cad68" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38699 } } peers { permanent_uuid: "140fc3d6b425431397bb971191bf6522" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 45277 } } peers { permanent_uuid: "5a28acc0c9294f36a7c9a01ad7aee8aa" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45507 } }
I20260808 18:32:39.671293 17583 catalog_manager.cc:5495] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522 reported cstate change: term changed from 0 to 1, leader changed from <none> to 140fc3d6b425431397bb971191bf6522 (127.29.78.130). New cstate: current_term: 1 leader_uuid: "140fc3d6b425431397bb971191bf6522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214b1244378d49d6b3e069ad254cad68" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "140fc3d6b425431397bb971191bf6522" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 45277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a28acc0c9294f36a7c9a01ad7aee8aa" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45507 } health_report { overall_health: UNKNOWN } } }
I20260808 18:32:39.689483 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:39.685957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.29.78.130:45277)
0808 18:32:39.686091 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:39.686095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:39.686109 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:32:39.686250 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:39.686253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:39.686301 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:39.686388 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:32:39.686662 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.687222 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.688038 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.688258 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.688346 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:39.688637 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:32:39.689042 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:32:39.689156 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:39.689239 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 18:32:39.689316 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 18:32:39.695076 17748 raft_consensus.cc:1261] T 6ffa1bfe6ca644cb819783776611a52a P 214b1244378d49d6b3e069ad254cad68 [term 1 FOLLOWER]: Refusing update from remote peer 140fc3d6b425431397bb971191bf6522: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:32:39.695081 18038 raft_consensus.cc:1261] T 6ffa1bfe6ca644cb819783776611a52a P 5a28acc0c9294f36a7c9a01ad7aee8aa [term 1 FOLLOWER]: Refusing update from remote peer 140fc3d6b425431397bb971191bf6522: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:32:39.695803 18118 consensus_queue.cc:1035] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "214b1244378d49d6b3e069ad254cad68" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:32:39.695994 18113 consensus_queue.cc:1035] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a28acc0c9294f36a7c9a01ad7aee8aa" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:32:39.712689 17583 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:32:39.712884 17583 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:32:39.715337 17583 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7cfb777711b4335908f4b20ddac226a: Sending DeleteTablet for 3 replicas of tablet 6ffa1bfe6ca644cb819783776611a52a
I20260808 18:32:39.716219 17873 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ffa1bfe6ca644cb819783776611a52a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:32:39 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:53274
I20260808 18:32:39.716223 17725 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ffa1bfe6ca644cb819783776611a52a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:32:39 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:33696
I20260808 18:32:39.716730 18134 tablet_replica.cc:330] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522: stopping tablet replica
I20260808 18:32:39.716956 18018 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ffa1bfe6ca644cb819783776611a52a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:32:39 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:36256
I20260808 18:32:39.717195 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 17654
I20260808 18:32:39.717299 18134 raft_consensus.cc:2229] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:32:39.717453 18136 tablet_replica.cc:330] T 6ffa1bfe6ca644cb819783776611a52a P 5a28acc0c9294f36a7c9a01ad7aee8aa: stopping tablet replica
I20260808 18:32:39.717866 18136 raft_consensus.cc:2229] T 6ffa1bfe6ca644cb819783776611a52a P 5a28acc0c9294f36a7c9a01ad7aee8aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:32:39.718165 18136 raft_consensus.cc:2258] T 6ffa1bfe6ca644cb819783776611a52a P 5a28acc0c9294f36a7c9a01ad7aee8aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:32:39.718538 18134 raft_consensus.cc:2258] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:32:39.720103 18136 ts_tablet_manager.cc:1905] T 6ffa1bfe6ca644cb819783776611a52a P 5a28acc0c9294f36a7c9a01ad7aee8aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:32:39.720103 18134 ts_tablet_manager.cc:1905] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:32:39.722852 18136 ts_tablet_manager.cc:1918] T 6ffa1bfe6ca644cb819783776611a52a P 5a28acc0c9294f36a7c9a01ad7aee8aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:32:39.722852 18134 ts_tablet_manager.cc:1918] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:32:39.722954 18136 log.cc:1196] T 6ffa1bfe6ca644cb819783776611a52a P 5a28acc0c9294f36a7c9a01ad7aee8aa: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/6ffa1bfe6ca644cb819783776611a52a
I20260808 18:32:39.723083 18134 log.cc:1196] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/6ffa1bfe6ca644cb819783776611a52a
I20260808 18:32:39.723394 18136 ts_tablet_manager.cc:1939] T 6ffa1bfe6ca644cb819783776611a52a P 5a28acc0c9294f36a7c9a01ad7aee8aa: Deleting consensus metadata
I20260808 18:32:39.723609 18134 ts_tablet_manager.cc:1939] T 6ffa1bfe6ca644cb819783776611a52a P 140fc3d6b425431397bb971191bf6522: Deleting consensus metadata
I20260808 18:32:39.724632 17572 catalog_manager.cc:4841] TS 5a28acc0c9294f36a7c9a01ad7aee8aa (127.29.78.131:45507): tablet 6ffa1bfe6ca644cb819783776611a52a (table test-table [id=a6810993010341ed8e37a7259c56ebce]) successfully deleted
I20260808 18:32:39.724859 17572 catalog_manager.cc:4841] TS 140fc3d6b425431397bb971191bf6522 (127.29.78.130:45277): tablet 6ffa1bfe6ca644cb819783776611a52a (table test-table [id=a6810993010341ed8e37a7259c56ebce]) successfully deleted
W20260808 18:32:39.729991 17570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:38699 (error 108)
I20260808 18:32:39.730902 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 17805
I20260808 18:32:39.731106 18104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:39.730801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:38699 (local address 127.0.0.1:33710)
0808 18:32:39.730951 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:39.731028 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:38699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260808 18:32:39.739001 17570 catalog_manager.cc:4584] TS 214b1244378d49d6b3e069ad254cad68 (127.29.78.129:38699): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6ffa1bfe6ca644cb819783776611a52a: Network error: Client connection negotiation failed: client connection to 127.29.78.129:38699: connect: Connection refused (error 111)
I20260808 18:32:39.742690 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 17949
I20260808 18:32:39.751739 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 17551
2026-08-08T18:32:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2326 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-08T18:32:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:32:39Z Disabled control of system clock
I20260808 18:32:39.786643 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:36631
--webserver_interface=127.29.78.190
--webserver_port=0
--builtin_ntp_servers=127.29.78.148:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:36631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 18:32:39.898027 18142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:39.898268 18142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:39.898303 18142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:39.902187 18142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:32:39.902235 18142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:39.902249 18142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:39.902266 18142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:32:39.902280 18142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:32:39.906814 18142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:36631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:36631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:39.907697 18142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:39.908912 18142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:39.917228 18148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:39.917220 18151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:39.917222 18149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:39.917621 18142 server_base.cc:1034] running on GCE node
I20260808 18:32:39.918308 18142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:39.919409 18142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:39.920609 18142 hybrid_clock.cc:648] HybridClock initialized: now 1786213959920582 us; error 39 us; skew 500 ppm
I20260808 18:32:39.922578 18142 webserver.cc:460] Webserver started at http://127.29.78.190:36209/ using document root <none> and password file <none>
I20260808 18:32:39.923157 18142 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:39.923251 18142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:39.923504 18142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:39.925522 18142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "5ca8648bfe644d348adedeb228018652"
format_stamp: "Formatted at 2026-08-08 18:32:39 on dist-test-slave-gp6n"
server_key: "2c89d1ec3568c3c9fac3845135e6c381"
server_key_iv: "ae5b01c89ebffb191c9d86a5b766caac"
server_key_version: "encryptionkey@0"
I20260808 18:32:39.926090 18142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "5ca8648bfe644d348adedeb228018652"
format_stamp: "Formatted at 2026-08-08 18:32:39 on dist-test-slave-gp6n"
server_key: "2c89d1ec3568c3c9fac3845135e6c381"
server_key_iv: "ae5b01c89ebffb191c9d86a5b766caac"
server_key_version: "encryptionkey@0"
I20260808 18:32:39.929754 18142 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 18:32:39.932212 18157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:39.933418 18142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:32:39.933568 18142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "5ca8648bfe644d348adedeb228018652"
format_stamp: "Formatted at 2026-08-08 18:32:39 on dist-test-slave-gp6n"
server_key: "2c89d1ec3568c3c9fac3845135e6c381"
server_key_iv: "ae5b01c89ebffb191c9d86a5b766caac"
server_key_version: "encryptionkey@0"
I20260808 18:32:39.933681 18142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:39.955121 18142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:39.955828 18142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:39.956035 18142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:39.962929 18142 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:36631
I20260808 18:32:39.962949 18211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:36631 every 8 connection(s)
I20260808 18:32:39.963881 18142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:32:39.966945 18213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:39.972323 18213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652: Bootstrap starting.
I20260808 18:32:39.972616 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 18142
I20260808 18:32:39.972764 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:32:39.973063 30010 external_mini_cluster.cc:1428] Setting key 06a3fbc61f42e9e3d0e9ae7b1fcce9ab
I20260808 18:32:39.975030 18213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:39.975788 18213 log.cc:824] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:39.977638 18213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652: No bootstrap required, opened a new log
I20260808 18:32:39.981215 18213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca8648bfe644d348adedeb228018652" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36631 } }
I20260808 18:32:39.981464 18213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:39.981519 18213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ca8648bfe644d348adedeb228018652, State: Initialized, Role: FOLLOWER
I20260808 18:32:39.981984 18213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca8648bfe644d348adedeb228018652" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36631 } }
I20260808 18:32:39.982146 18213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:32:39.982234 18213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:32:39.982357 18213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:39.982707 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:39.974510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.29.78.190:36631)
0808 18:32:39.975118 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:39.975156 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:39.975205 (+ 49us) server_negotiation.cc:407] Connection header received
0808 18:32:39.975822 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:39.975839 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:39.976314 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:39.976652 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:39.977077 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.978152 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.979138 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:39.979627 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:39.979804 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:39.980744 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:39.980748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:39.980760 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:39.981141 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:39.981196 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:39.981215 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:39.981325 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:39.981474 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 18:32:39.981771 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":126,"threads_started":1}
I20260808 18:32:39.983484 18213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca8648bfe644d348adedeb228018652" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36631 } }
I20260808 18:32:39.983821 18213 leader_election.cc:302] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ca8648bfe644d348adedeb228018652; no voters:
I20260808 18:32:39.984169 18213 leader_election.cc:288] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:32:39.984237 18218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:32:39.984467 18218 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652 [term 1 LEADER]: Becoming Leader. State: Replica: 5ca8648bfe644d348adedeb228018652, State: Running, Role: LEADER
I20260808 18:32:39.984754 18218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca8648bfe644d348adedeb228018652" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36631 } }
I20260808 18:32:39.985450 18213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:32:39.985925 18219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ca8648bfe644d348adedeb228018652. Latest consensus state: current_term: 1 leader_uuid: "5ca8648bfe644d348adedeb228018652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca8648bfe644d348adedeb228018652" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36631 } } }
I20260808 18:32:39.985983 18220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ca8648bfe644d348adedeb228018652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca8648bfe644d348adedeb228018652" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36631 } } }
I20260808 18:32:39.986166 18220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652 [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:39.986202 18219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652 [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:39.986622 18224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:32:39.989817 18224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:32:39.995374 18224 catalog_manager.cc:1347] Generated new cluster ID: 8a738c3cbe144de6bd09ef760364d31b
I20260808 18:32:39.995447 18224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:32:40.007612 18224 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:32:40.008527 18224 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:32:40.017633 18224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652: Generated new TSK 0
I20260808 18:32:40.018471 18224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:32:40.025084 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36631
--builtin_ntp_servers=127.29.78.148:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 18:32:40.142048 18238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:40.142298 18238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:40.142335 18238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:40.146211 18238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:40.146262 18238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:40.146315 18238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:32:40.151144 18238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:40.152092 18238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:40.153455 18238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:40.162744 18244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:40.162711 18245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:40.162711 18248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:40.163239 18238 server_base.cc:1034] running on GCE node
I20260808 18:32:40.163596 18238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:40.164186 18238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:40.165349 18238 hybrid_clock.cc:648] HybridClock initialized: now 1786213960165330 us; error 30 us; skew 500 ppm
I20260808 18:32:40.167474 18238 webserver.cc:460] Webserver started at http://127.29.78.129:37613/ using document root <none> and password file <none>
I20260808 18:32:40.168092 18238 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:40.168185 18238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:40.168406 18238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:40.170718 18238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "91dec12145b845f78a1a42b0014cd277"
format_stamp: "Formatted at 2026-08-08 18:32:40 on dist-test-slave-gp6n"
server_key: "6c0e2c66398ec848835965e1bca612bf"
server_key_iv: "9132abf82a50cda24c6a7699cbc9c42b"
server_key_version: "encryptionkey@0"
I20260808 18:32:40.171289 18238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "91dec12145b845f78a1a42b0014cd277"
format_stamp: "Formatted at 2026-08-08 18:32:40 on dist-test-slave-gp6n"
server_key: "6c0e2c66398ec848835965e1bca612bf"
server_key_iv: "9132abf82a50cda24c6a7699cbc9c42b"
server_key_version: "encryptionkey@0"
I20260808 18:32:40.175072 18238 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 18:32:40.177470 18255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:40.178722 18238 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:32:40.178853 18238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "91dec12145b845f78a1a42b0014cd277"
format_stamp: "Formatted at 2026-08-08 18:32:40 on dist-test-slave-gp6n"
server_key: "6c0e2c66398ec848835965e1bca612bf"
server_key_iv: "9132abf82a50cda24c6a7699cbc9c42b"
server_key_version: "encryptionkey@0"
I20260808 18:32:40.178962 18238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:40.195102 18238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:40.195751 18238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:40.195945 18238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:40.196511 18238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:32:40.197422 18238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:32:40.197490 18238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:40.197558 18238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:32:40.197598 18238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:40.206964 18238 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:35557
I20260808 18:32:40.207006 18375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:35557 every 8 connection(s)
I20260808 18:32:40.207830 18238 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:32:40.213276 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 18238
I20260808 18:32:40.213428 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:32:40.213738 30010 external_mini_cluster.cc:1428] Setting key 4624064c13a4e262a9734fcb968c3895
I20260808 18:32:40.215816 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36631
--builtin_ntp_servers=127.29.78.148:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 18:32:40.218086 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:40.209666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:57535 (local address 127.29.78.190:36631)
0808 18:32:40.209809 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:40.209814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:40.210934 (+ 1120us) server_negotiation.cc:407] Connection header received
0808 18:32:40.212262 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:40.212265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:40.212310 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:40.212393 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:40.213431 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.214058 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.215575 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.215792 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.215874 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:40.217310 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:40.217316 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:40.217323 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:40.217341 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:40.217366 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:40.217379 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:40.217461 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:40.217685 (+ 224us) server_negotiation.cc:299] Negotiation successful
0808 18:32:40.217785 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 18:32:40.219043 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:40.210113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36631 (local address 127.29.78.129:57535)
0808 18:32:40.210795 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:40.210835 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:40.212071 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:40.212586 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:40.212592 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:40.212674 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:32:40.213260 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:40.213281 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.214216 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.214219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:40.215447 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:40.215459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.215965 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.215969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:40.216202 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:40.216962 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:32:40.216966 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:32:40.217023 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:32:40.217106 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:32:40.217481 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:32:40.217493 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:32:40.217502 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 18:32:40.217645 (+ 143us) client_negotiation.cc:240] Negotiation successful
0808 18:32:40.217955 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":108,"threads_started":1}
I20260808 18:32:40.220520 18376 heartbeater.cc:347] Connected to a master server at 127.29.78.190:36631
I20260808 18:32:40.220834 18376 heartbeater.cc:464] Registering TS with master...
I20260808 18:32:40.222162 18376 heartbeater.cc:511] Master 127.29.78.190:36631 requested a full tablet report, sending...
I20260808 18:32:40.230374 18174 ts_manager.cc:194] Registered new tserver with Master: 91dec12145b845f78a1a42b0014cd277 (127.29.78.129:35557)
I20260808 18:32:40.232302 18174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.129:57535
W20260808 18:32:40.331490 18379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:40.331733 18379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:40.331772 18379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:40.335623 18379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:40.335668 18379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:40.335722 18379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:32:40.340477 18379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:40.341444 18379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:40.342756 18379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:32:40.352402 18379 server_base.cc:1034] running on GCE node
W20260808 18:32:40.352603 18384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:40.352628 18385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:40.352638 18387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:40.353194 18379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:40.353875 18379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:40.355064 18379 hybrid_clock.cc:648] HybridClock initialized: now 1786213960355045 us; error 30 us; skew 500 ppm
I20260808 18:32:40.357033 18379 webserver.cc:460] Webserver started at http://127.29.78.130:35685/ using document root <none> and password file <none>
I20260808 18:32:40.357631 18379 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:40.357728 18379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:40.357969 18379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:40.360132 18379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "4e33d84aa7964e04836d0c9cca43534c"
format_stamp: "Formatted at 2026-08-08 18:32:40 on dist-test-slave-gp6n"
server_key: "fbca35628940d180e0a5e21e0b9bf87e"
server_key_iv: "9deffa94d3588b56b22c11337356b875"
server_key_version: "encryptionkey@0"
I20260808 18:32:40.360708 18379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "4e33d84aa7964e04836d0c9cca43534c"
format_stamp: "Formatted at 2026-08-08 18:32:40 on dist-test-slave-gp6n"
server_key: "fbca35628940d180e0a5e21e0b9bf87e"
server_key_iv: "9deffa94d3588b56b22c11337356b875"
server_key_version: "encryptionkey@0"
I20260808 18:32:40.364549 18379 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260808 18:32:40.366999 18393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:40.368214 18379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:32:40.368348 18379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "4e33d84aa7964e04836d0c9cca43534c"
format_stamp: "Formatted at 2026-08-08 18:32:40 on dist-test-slave-gp6n"
server_key: "fbca35628940d180e0a5e21e0b9bf87e"
server_key_iv: "9deffa94d3588b56b22c11337356b875"
server_key_version: "encryptionkey@0"
I20260808 18:32:40.368464 18379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:40.381064 18379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:40.381699 18379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:40.381948 18379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:40.382570 18379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:32:40.383536 18379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:32:40.383608 18379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:40.383680 18379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:32:40.383723 18379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:40.392987 18379 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:35455
I20260808 18:32:40.393006 18525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:35455 every 8 connection(s)
I20260808 18:32:40.393847 18379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:32:40.394270 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 18379
I20260808 18:32:40.394433 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:32:40.394784 30010 external_mini_cluster.cc:1428] Setting key d1e01f48a36afbaaca8fc83421b1d254
I20260808 18:32:40.397063 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36631
--builtin_ntp_servers=127.29.78.148:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 18:32:40.403998 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:40.395888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:50249 (local address 127.29.78.190:36631)
0808 18:32:40.396021 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:40.396026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:40.397303 (+ 1277us) server_negotiation.cc:407] Connection header received
0808 18:32:40.398884 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:40.398887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:40.398935 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:40.399032 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:40.400250 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.400805 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.401968 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.402196 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.402311 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:40.403494 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:40.403497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:40.403499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:40.403511 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:40.403530 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:40.403539 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:40.403618 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:40.403784 (+ 166us) server_negotiation.cc:299] Negotiation successful
0808 18:32:40.403849 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 18:32:40.404843 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:40.396417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36631 (local address 127.29.78.130:50249)
0808 18:32:40.397144 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:40.397204 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:40.398612 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:40.399231 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:40.399241 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:40.399357 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:32:40.400079 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:40.400097 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.400952 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.400955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:40.401828 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:40.401837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.402350 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.402354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:40.402558 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:40.403211 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:32:40.403213 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:32:40.403252 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:32:40.403314 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:32:40.403632 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:32:40.403642 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:32:40.403648 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 18:32:40.403783 (+ 135us) client_negotiation.cc:240] Negotiation successful
0808 18:32:40.404069 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":159,"threads_started":1}
I20260808 18:32:40.406024 18526 heartbeater.cc:347] Connected to a master server at 127.29.78.190:36631
I20260808 18:32:40.406270 18526 heartbeater.cc:464] Registering TS with master...
I20260808 18:32:40.406862 18526 heartbeater.cc:511] Master 127.29.78.190:36631 requested a full tablet report, sending...
I20260808 18:32:40.407951 18174 ts_manager.cc:194] Registered new tserver with Master: 4e33d84aa7964e04836d0c9cca43534c (127.29.78.130:35455)
I20260808 18:32:40.408839 18174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.130:50249
W20260808 18:32:40.511241 18529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:40.511508 18529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:40.511548 18529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:40.515578 18529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:40.515630 18529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:40.515681 18529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:32:40.520260 18529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:40.521239 18529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:40.522606 18529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:40.532068 18535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:40.532084 18537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:40.532078 18534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:40.532357 18529 server_base.cc:1034] running on GCE node
I20260808 18:32:40.532931 18529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:40.533586 18529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:40.534781 18529 hybrid_clock.cc:648] HybridClock initialized: now 1786213960534757 us; error 33 us; skew 500 ppm
I20260808 18:32:40.536785 18529 webserver.cc:460] Webserver started at http://127.29.78.131:42677/ using document root <none> and password file <none>
I20260808 18:32:40.537444 18529 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:40.537556 18529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:40.537851 18529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:40.540089 18529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "1fa564a3d6e3430eb7d536986cd8023a"
format_stamp: "Formatted at 2026-08-08 18:32:40 on dist-test-slave-gp6n"
server_key: "94b749a3f632e511f26bef39c313ee91"
server_key_iv: "9d4505260658a43bb59668d3d3002dcc"
server_key_version: "encryptionkey@0"
I20260808 18:32:40.540696 18529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "1fa564a3d6e3430eb7d536986cd8023a"
format_stamp: "Formatted at 2026-08-08 18:32:40 on dist-test-slave-gp6n"
server_key: "94b749a3f632e511f26bef39c313ee91"
server_key_iv: "9d4505260658a43bb59668d3d3002dcc"
server_key_version: "encryptionkey@0"
I20260808 18:32:40.544589 18529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:32:40.547338 18545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:40.548818 18529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 18:32:40.548964 18529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "1fa564a3d6e3430eb7d536986cd8023a"
format_stamp: "Formatted at 2026-08-08 18:32:40 on dist-test-slave-gp6n"
server_key: "94b749a3f632e511f26bef39c313ee91"
server_key_iv: "9d4505260658a43bb59668d3d3002dcc"
server_key_version: "encryptionkey@0"
I20260808 18:32:40.549084 18529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:40.565696 18529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:40.566314 18529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:40.566542 18529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:40.567091 18529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:32:40.568018 18529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:32:40.568082 18529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:40.568146 18529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:32:40.568193 18529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:40.577090 18529 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:37287
I20260808 18:32:40.577163 18670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:37287 every 8 connection(s)
I20260808 18:32:40.578025 18529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:32:40.584565 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 18529
I20260808 18:32:40.584690 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:32:40.584954 30010 external_mini_cluster.cc:1428] Setting key be9d6389dc18cf3bd841c513e939c4bb
I20260808 18:32:40.588088 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:40.579806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:58295 (local address 127.29.78.190:36631)
0808 18:32:40.579935 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:40.579950 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:40.580998 (+ 1048us) server_negotiation.cc:407] Connection header received
0808 18:32:40.582602 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:40.582604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:40.582654 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:40.582733 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:40.583715 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.584366 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.586023 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.586298 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.586383 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:40.587666 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:40.587669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:40.587671 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:40.587682 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:40.587701 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:40.587709 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:40.587770 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:40.587922 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 18:32:40.587973 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 18:32:40.589037 18674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:40.580143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36631 (local address 127.29.78.131:58295)
0808 18:32:40.580822 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:40.580866 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:40.582235 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:40.582905 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:40.582927 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:40.583017 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:32:40.583562 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:40.583581 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.584503 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.584505 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:40.585831 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:40.585844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.586492 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.586498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:40.586670 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:40.587307 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:32:40.587309 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:32:40.587351 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:32:40.587451 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:32:40.587788 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:32:40.587795 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:32:40.587800 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 18:32:40.587903 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 18:32:40.588133 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":104,"threads_started":1}
I20260808 18:32:40.590111 18671 heartbeater.cc:347] Connected to a master server at 127.29.78.190:36631
I20260808 18:32:40.590312 18671 heartbeater.cc:464] Registering TS with master...
I20260808 18:32:40.590839 18671 heartbeater.cc:511] Master 127.29.78.190:36631 requested a full tablet report, sending...
I20260808 18:32:40.591784 18174 ts_manager.cc:194] Registered new tserver with Master: 1fa564a3d6e3430eb7d536986cd8023a (127.29.78.131:37287)
I20260808 18:32:40.592289 18174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.131:58295
I20260808 18:32:40.599164 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:32:40.604715 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:40.601149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35376 (local address 127.29.78.190:36631)
0808 18:32:40.601270 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:40.601273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:40.601401 (+ 128us) server_negotiation.cc:407] Connection header received
0808 18:32:40.601499 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:40.601501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:40.601553 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:40.601630 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:40.601882 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.602376 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.603219 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.603424 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.603497 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:40.604315 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:40.604318 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:40.604320 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:40.604330 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:40.604346 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:40.604353 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:40.604438 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:40.604580 (+ 142us) server_negotiation.cc:299] Negotiation successful
0808 18:32:40.604612 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 18:32:40.607574 18174 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:32:40.610606 18174 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:32:40.624734 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:40.618751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:37287 (local address 127.0.0.1:49066)
0808 18:32:40.619691 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:40.619742 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:40.619955 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:40.620776 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:40.620782 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:40.620806 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:40.620944 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:40.620951 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.621953 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.621957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:40.623409 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:40.623418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.624200 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.624204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:40.624307 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:40.624353 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 18:32:40.624460 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 18:32:40.624575 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":37,"spinlock_wait_cycles":4736,"thread_start_us":92,"threads_started":1}
I20260808 18:32:40.624738 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:40.618875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49066 (local address 127.29.78.131:37287)
0808 18:32:40.619248 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:40.619262 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:40.619836 (+ 574us) server_negotiation.cc:407] Connection header received
0808 18:32:40.620160 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:40.620166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:40.620420 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:40.620562 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:40.621088 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.621825 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.623542 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.624075 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.624201 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:40.624394 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:40.624484 (+ 90us) server_negotiation.cc:299] Negotiation successful
0808 18:32:40.624575 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":196,"threads_started":1}
I20260808 18:32:40.625116 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:40.619548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:35455 (local address 127.0.0.1:54774)
0808 18:32:40.620569 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:40.620583 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:40.620675 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:40.621104 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:40.621107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:40.621122 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:40.621193 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:40.621201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.622235 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.622239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:40.623420 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:40.623429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.624692 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.624695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:40.624783 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:40.624805 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 18:32:40.624875 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 18:32:40.624939 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"mutex_wait_us":2,"thread_start_us":106,"threads_started":1}
I20260808 18:32:40.625566 18690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:40.620093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54774 (local address 127.29.78.130:35455)
0808 18:32:40.620454 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:40.620465 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:40.620677 (+ 212us) server_negotiation.cc:407] Connection header received
0808 18:32:40.620782 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:40.620787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:40.620962 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:40.621104 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:40.621286 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.622108 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.623576 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.624353 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.624450 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:40.624624 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:40.625305 (+ 681us) server_negotiation.cc:299] Negotiation successful
0808 18:32:40.625412 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":151,"threads_started":1}
I20260808 18:32:40.625777 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:40.618855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:35557 (local address 127.0.0.1:53094)
0808 18:32:40.619810 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:40.619823 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:40.619954 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:40.620833 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:40.620836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:40.620854 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:40.620956 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:40.620964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.623177 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.623180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:40.624283 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:40.624295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.625423 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.625426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:40.625510 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:40.625532 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 18:32:40.625596 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 18:32:40.625657 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"spinlock_wait_cycles":7424,"thread_start_us":97,"threads_started":1}
I20260808 18:32:40.626772 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:40.619509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53094 (local address 127.29.78.129:35557)
0808 18:32:40.620291 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:40.620303 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:40.620328 (+ 25us) server_negotiation.cc:407] Connection header received
0808 18:32:40.620451 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:40.620460 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:40.620661 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:40.620866 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:40.621955 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.623059 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.624440 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.625244 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.625391 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:40.626441 (+ 1050us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:40.626504 (+ 63us) server_negotiation.cc:299] Negotiation successful
0808 18:32:40.626601 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":141,"threads_started":1}
I20260808 18:32:40.628177 18599 tablet_service.cc:1467] Processing CreateTablet for tablet 2dce3e41fef049ec86dd8a8c8e87d574 (DEFAULT_TABLE table=test-table [id=2a04d69412ee4d5ba0c3bb70edb90905]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:32:40.628525 18452 tablet_service.cc:1467] Processing CreateTablet for tablet 2dce3e41fef049ec86dd8a8c8e87d574 (DEFAULT_TABLE table=test-table [id=2a04d69412ee4d5ba0c3bb70edb90905]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:32:40.628948 18307 tablet_service.cc:1467] Processing CreateTablet for tablet 2dce3e41fef049ec86dd8a8c8e87d574 (DEFAULT_TABLE table=test-table [id=2a04d69412ee4d5ba0c3bb70edb90905]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:32:40.629326 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dce3e41fef049ec86dd8a8c8e87d574. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:40.629470 18452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dce3e41fef049ec86dd8a8c8e87d574. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:40.629995 18307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dce3e41fef049ec86dd8a8c8e87d574. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:40.635835 18691 tablet_bootstrap.cc:492] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277: Bootstrap starting.
I20260808 18:32:40.636492 18692 tablet_bootstrap.cc:492] T 2dce3e41fef049ec86dd8a8c8e87d574 P 1fa564a3d6e3430eb7d536986cd8023a: Bootstrap starting.
I20260808 18:32:40.637221 18693 tablet_bootstrap.cc:492] T 2dce3e41fef049ec86dd8a8c8e87d574 P 4e33d84aa7964e04836d0c9cca43534c: Bootstrap starting.
I20260808 18:32:40.638863 18692 tablet_bootstrap.cc:654] T 2dce3e41fef049ec86dd8a8c8e87d574 P 1fa564a3d6e3430eb7d536986cd8023a: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:40.638863 18691 tablet_bootstrap.cc:654] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:40.639156 18693 tablet_bootstrap.cc:654] T 2dce3e41fef049ec86dd8a8c8e87d574 P 4e33d84aa7964e04836d0c9cca43534c: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:40.639750 18691 log.cc:824] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:40.639750 18692 log.cc:824] T 2dce3e41fef049ec86dd8a8c8e87d574 P 1fa564a3d6e3430eb7d536986cd8023a: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:40.639803 18693 log.cc:824] T 2dce3e41fef049ec86dd8a8c8e87d574 P 4e33d84aa7964e04836d0c9cca43534c: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:40.641613 18692 tablet_bootstrap.cc:492] T 2dce3e41fef049ec86dd8a8c8e87d574 P 1fa564a3d6e3430eb7d536986cd8023a: No bootstrap required, opened a new log
I20260808 18:32:40.641786 18693 tablet_bootstrap.cc:492] T 2dce3e41fef049ec86dd8a8c8e87d574 P 4e33d84aa7964e04836d0c9cca43534c: No bootstrap required, opened a new log
I20260808 18:32:40.641829 18692 ts_tablet_manager.cc:1397] T 2dce3e41fef049ec86dd8a8c8e87d574 P 1fa564a3d6e3430eb7d536986cd8023a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260808 18:32:40.641886 18691 tablet_bootstrap.cc:492] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277: No bootstrap required, opened a new log
I20260808 18:32:40.641971 18693 ts_tablet_manager.cc:1397] T 2dce3e41fef049ec86dd8a8c8e87d574 P 4e33d84aa7964e04836d0c9cca43534c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 18:32:40.642035 18691 ts_tablet_manager.cc:1397] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:32:40.645077 18692 raft_consensus.cc:348] T 2dce3e41fef049ec86dd8a8c8e87d574 P 1fa564a3d6e3430eb7d536986cd8023a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa564a3d6e3430eb7d536986cd8023a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37287 } } peers { permanent_uuid: "91dec12145b845f78a1a42b0014cd277" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35557 } } peers { permanent_uuid: "4e33d84aa7964e04836d0c9cca43534c" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35455 } }
I20260808 18:32:40.645329 18692 raft_consensus.cc:374] T 2dce3e41fef049ec86dd8a8c8e87d574 P 1fa564a3d6e3430eb7d536986cd8023a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:40.645412 18692 raft_consensus.cc:729] T 2dce3e41fef049ec86dd8a8c8e87d574 P 1fa564a3d6e3430eb7d536986cd8023a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa564a3d6e3430eb7d536986cd8023a, State: Initialized, Role: FOLLOWER
I20260808 18:32:40.645915 18692 consensus_queue.cc:260] T 2dce3e41fef049ec86dd8a8c8e87d574 P 1fa564a3d6e3430eb7d536986cd8023a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa564a3d6e3430eb7d536986cd8023a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37287 } } peers { permanent_uuid: "91dec12145b845f78a1a42b0014cd277" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35557 } } peers { permanent_uuid: "4e33d84aa7964e04836d0c9cca43534c" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35455 } }
I20260808 18:32:40.646317 18691 raft_consensus.cc:348] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa564a3d6e3430eb7d536986cd8023a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37287 } } peers { permanent_uuid: "91dec12145b845f78a1a42b0014cd277" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35557 } } peers { permanent_uuid: "4e33d84aa7964e04836d0c9cca43534c" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35455 } }
I20260808 18:32:40.646317 18693 raft_consensus.cc:348] T 2dce3e41fef049ec86dd8a8c8e87d574 P 4e33d84aa7964e04836d0c9cca43534c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa564a3d6e3430eb7d536986cd8023a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37287 } } peers { permanent_uuid: "91dec12145b845f78a1a42b0014cd277" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35557 } } peers { permanent_uuid: "4e33d84aa7964e04836d0c9cca43534c" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35455 } }
I20260808 18:32:40.646598 18693 raft_consensus.cc:374] T 2dce3e41fef049ec86dd8a8c8e87d574 P 4e33d84aa7964e04836d0c9cca43534c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:40.646605 18671 heartbeater.cc:503] Master 127.29.78.190:36631 was elected leader, sending a full tablet report...
I20260808 18:32:40.646629 18691 raft_consensus.cc:374] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:40.646682 18693 raft_consensus.cc:729] T 2dce3e41fef049ec86dd8a8c8e87d574 P 4e33d84aa7964e04836d0c9cca43534c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e33d84aa7964e04836d0c9cca43534c, State: Initialized, Role: FOLLOWER
I20260808 18:32:40.647006 18692 ts_tablet_manager.cc:1428] T 2dce3e41fef049ec86dd8a8c8e87d574 P 1fa564a3d6e3430eb7d536986cd8023a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 18:32:40.647156 18693 consensus_queue.cc:260] T 2dce3e41fef049ec86dd8a8c8e87d574 P 4e33d84aa7964e04836d0c9cca43534c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa564a3d6e3430eb7d536986cd8023a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37287 } } peers { permanent_uuid: "91dec12145b845f78a1a42b0014cd277" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35557 } } peers { permanent_uuid: "4e33d84aa7964e04836d0c9cca43534c" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35455 } }
I20260808 18:32:40.646714 18691 raft_consensus.cc:729] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91dec12145b845f78a1a42b0014cd277, State: Initialized, Role: FOLLOWER
I20260808 18:32:40.647706 18691 consensus_queue.cc:260] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa564a3d6e3430eb7d536986cd8023a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37287 } } peers { permanent_uuid: "91dec12145b845f78a1a42b0014cd277" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35557 } } peers { permanent_uuid: "4e33d84aa7964e04836d0c9cca43534c" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35455 } }
I20260808 18:32:40.648279 18693 ts_tablet_manager.cc:1428] T 2dce3e41fef049ec86dd8a8c8e87d574 P 4e33d84aa7964e04836d0c9cca43534c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 18:32:40.648483 18376 heartbeater.cc:503] Master 127.29.78.190:36631 was elected leader, sending a full tablet report...
I20260808 18:32:40.648900 18691 ts_tablet_manager.cc:1428] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 18:32:40.648681 18526 heartbeater.cc:503] Master 127.29.78.190:36631 was elected leader, sending a full tablet report...
W20260808 18:32:40.709038 18377 tablet.cc:2369] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 18:32:40.829080 18673 tablet.cc:2369] T 2dce3e41fef049ec86dd8a8c8e87d574 P 1fa564a3d6e3430eb7d536986cd8023a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 18:32:40.895949 18527 tablet.cc:2369] T 2dce3e41fef049ec86dd8a8c8e87d574 P 4e33d84aa7964e04836d0c9cca43534c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:32:40.929823 18700 raft_consensus.cc:482] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:32:40.930084 18700 raft_consensus.cc:504] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa564a3d6e3430eb7d536986cd8023a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37287 } } peers { permanent_uuid: "91dec12145b845f78a1a42b0014cd277" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35557 } } peers { permanent_uuid: "4e33d84aa7964e04836d0c9cca43534c" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35455 } }
I20260808 18:32:40.931329 18700 leader_election.cc:288] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fa564a3d6e3430eb7d536986cd8023a (127.29.78.131:37287), 4e33d84aa7964e04836d0c9cca43534c (127.29.78.130:35455)
I20260808 18:32:40.935495 18690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:40.931774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:52173 (local address 127.29.78.130:35455)
0808 18:32:40.931918 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:40.931923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:40.932292 (+ 369us) server_negotiation.cc:407] Connection header received
0808 18:32:40.932421 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:40.932424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:40.932476 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:40.932582 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:40.932874 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.933381 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.934520 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.934901 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.934997 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:40.935077 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:40.935269 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 18:32:40.935347 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 18:32:40.935518 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:40.931795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:47217 (local address 127.29.78.131:37287)
0808 18:32:40.932035 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:40.932039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:40.932082 (+ 43us) server_negotiation.cc:407] Connection header received
0808 18:32:40.932252 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:40.932257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:40.932317 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:40.932398 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:40.932702 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.933278 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.934518 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.934933 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.935071 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:40.935122 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:40.935333 (+ 211us) server_negotiation.cc:299] Negotiation successful
0808 18:32:40.935413 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260808 18:32:40.935515 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:40.931641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:35455 (local address 127.29.78.129:52173)
0808 18:32:40.932158 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:40.932194 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:40.932297 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:40.932600 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:40.932602 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:40.932617 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:40.932758 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:40.932765 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.933491 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.933493 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:40.934332 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:40.934341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.935024 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.935031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:40.935122 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:40.935150 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 18:32:40.935221 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 18:32:40.935300 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":243,"thread_start_us":55,"threads_started":1}
I20260808 18:32:40.935519 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:40.931641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:37287 (local address 127.29.78.129:47217)
0808 18:32:40.932004 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:40.932017 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:40.932129 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:40.932426 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:40.932429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:40.932446 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:40.932592 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:40.932599 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.933395 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.933400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:40.934336 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:40.934345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.935099 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.935104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:40.935183 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:40.935221 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 18:32:40.935301 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 18:32:40.935364 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":36,"thread_start_us":114,"threads_started":1}
I20260808 18:32:40.936144 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dce3e41fef049ec86dd8a8c8e87d574" candidate_uuid: "91dec12145b845f78a1a42b0014cd277" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e33d84aa7964e04836d0c9cca43534c" is_pre_election: true
I20260808 18:32:40.936450 18474 raft_consensus.cc:2454] T 2dce3e41fef049ec86dd8a8c8e87d574 P 4e33d84aa7964e04836d0c9cca43534c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91dec12145b845f78a1a42b0014cd277 in term 0.
I20260808 18:32:40.936414 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dce3e41fef049ec86dd8a8c8e87d574" candidate_uuid: "91dec12145b845f78a1a42b0014cd277" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa564a3d6e3430eb7d536986cd8023a" is_pre_election: true
I20260808 18:32:40.936755 18621 raft_consensus.cc:2454] T 2dce3e41fef049ec86dd8a8c8e87d574 P 1fa564a3d6e3430eb7d536986cd8023a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91dec12145b845f78a1a42b0014cd277 in term 0.
I20260808 18:32:40.937023 18257 leader_election.cc:302] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e33d84aa7964e04836d0c9cca43534c, 91dec12145b845f78a1a42b0014cd277; no voters:
I20260808 18:32:40.937322 18700 raft_consensus.cc:2781] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:32:40.937417 18700 raft_consensus.cc:482] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:32:40.937474 18700 raft_consensus.cc:3037] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:40.938673 18700 raft_consensus.cc:504] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa564a3d6e3430eb7d536986cd8023a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37287 } } peers { permanent_uuid: "91dec12145b845f78a1a42b0014cd277" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35557 } } peers { permanent_uuid: "4e33d84aa7964e04836d0c9cca43534c" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35455 } }
I20260808 18:32:40.939071 18700 leader_election.cc:288] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277 [CANDIDATE]: Term 1 election: Requested vote from peers 1fa564a3d6e3430eb7d536986cd8023a (127.29.78.131:37287), 4e33d84aa7964e04836d0c9cca43534c (127.29.78.130:35455)
I20260808 18:32:40.939334 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dce3e41fef049ec86dd8a8c8e87d574" candidate_uuid: "91dec12145b845f78a1a42b0014cd277" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa564a3d6e3430eb7d536986cd8023a"
I20260808 18:32:40.939471 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dce3e41fef049ec86dd8a8c8e87d574" candidate_uuid: "91dec12145b845f78a1a42b0014cd277" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e33d84aa7964e04836d0c9cca43534c"
I20260808 18:32:40.939462 18621 raft_consensus.cc:3037] T 2dce3e41fef049ec86dd8a8c8e87d574 P 1fa564a3d6e3430eb7d536986cd8023a [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:40.939630 18474 raft_consensus.cc:3037] T 2dce3e41fef049ec86dd8a8c8e87d574 P 4e33d84aa7964e04836d0c9cca43534c [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:40.940625 18621 raft_consensus.cc:2454] T 2dce3e41fef049ec86dd8a8c8e87d574 P 1fa564a3d6e3430eb7d536986cd8023a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91dec12145b845f78a1a42b0014cd277 in term 1.
I20260808 18:32:40.940727 18474 raft_consensus.cc:2454] T 2dce3e41fef049ec86dd8a8c8e87d574 P 4e33d84aa7964e04836d0c9cca43534c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91dec12145b845f78a1a42b0014cd277 in term 1.
I20260808 18:32:40.940968 18256 leader_election.cc:302] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa564a3d6e3430eb7d536986cd8023a, 91dec12145b845f78a1a42b0014cd277; no voters:
I20260808 18:32:40.941140 18700 raft_consensus.cc:2781] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:32:40.941390 18700 raft_consensus.cc:686] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277 [term 1 LEADER]: Becoming Leader. State: Replica: 91dec12145b845f78a1a42b0014cd277, State: Running, Role: LEADER
I20260808 18:32:40.941731 18700 consensus_queue.cc:237] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa564a3d6e3430eb7d536986cd8023a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37287 } } peers { permanent_uuid: "91dec12145b845f78a1a42b0014cd277" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35557 } } peers { permanent_uuid: "4e33d84aa7964e04836d0c9cca43534c" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35455 } }
I20260808 18:32:40.944698 18174 catalog_manager.cc:5495] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91dec12145b845f78a1a42b0014cd277 (127.29.78.129). New cstate: current_term: 1 leader_uuid: "91dec12145b845f78a1a42b0014cd277" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa564a3d6e3430eb7d536986cd8023a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91dec12145b845f78a1a42b0014cd277" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 35557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e33d84aa7964e04836d0c9cca43534c" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35455 } health_report { overall_health: UNKNOWN } } }
I20260808 18:32:40.988727 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:40.984510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53098 (local address 127.29.78.129:35557)
0808 18:32:40.984673 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:40.984677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:40.984703 (+ 26us) server_negotiation.cc:407] Connection header received
0808 18:32:40.984840 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:40.984843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:40.984926 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:40.985072 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:32:40.985378 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.986047 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.987099 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:40.987318 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:40.987411 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:40.987785 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:32:40.988305 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:32:40.988463 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:40.988583 (+ 120us) server_negotiation.cc:299] Negotiation successful
0808 18:32:40.988617 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 18:32:40.994635 18621 raft_consensus.cc:1261] T 2dce3e41fef049ec86dd8a8c8e87d574 P 1fa564a3d6e3430eb7d536986cd8023a [term 1 FOLLOWER]: Refusing update from remote peer 91dec12145b845f78a1a42b0014cd277: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:32:40.994694 18474 raft_consensus.cc:1261] T 2dce3e41fef049ec86dd8a8c8e87d574 P 4e33d84aa7964e04836d0c9cca43534c [term 1 FOLLOWER]: Refusing update from remote peer 91dec12145b845f78a1a42b0014cd277: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:32:40.995380 18700 consensus_queue.cc:1035] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fa564a3d6e3430eb7d536986cd8023a" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:32:40.995585 18703 consensus_queue.cc:1035] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e33d84aa7964e04836d0c9cca43534c" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:32:41.003595 18707 mvcc.cc:205] Tried to move back new op lower bound from 7316332384226652160 to 7316332384019820544. Current Snapshot: MvccSnapshot[applied={T|T < 7316332384226652160}]
I20260808 18:32:41.005755 18709 mvcc.cc:205] Tried to move back new op lower bound from 7316332384226652160 to 7316332384019820544. Current Snapshot: MvccSnapshot[applied={T|T < 7316332384226652160}]
I20260808 18:32:41.011238 18172 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:32:41.011415 18172 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:32:41.013922 18172 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ca8648bfe644d348adedeb228018652: Sending DeleteTablet for 3 replicas of tablet 2dce3e41fef049ec86dd8a8c8e87d574
I20260808 18:32:41.014745 18599 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dce3e41fef049ec86dd8a8c8e87d574 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:32:41 UTC) from {username='slave'} at 127.0.0.1:49066
I20260808 18:32:41.014792 18307 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dce3e41fef049ec86dd8a8c8e87d574 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:32:41 UTC) from {username='slave'} at 127.0.0.1:53094
I20260808 18:32:41.014914 18452 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dce3e41fef049ec86dd8a8c8e87d574 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:32:41 UTC) from {username='slave'} at 127.0.0.1:54774
I20260808 18:32:41.015287 18719 tablet_replica.cc:330] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277: stopping tablet replica
I20260808 18:32:41.015486 18718 tablet_replica.cc:330] T 2dce3e41fef049ec86dd8a8c8e87d574 P 1fa564a3d6e3430eb7d536986cd8023a: stopping tablet replica
I20260808 18:32:41.015622 18719 raft_consensus.cc:2229] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:32:41.015883 18718 raft_consensus.cc:2229] T 2dce3e41fef049ec86dd8a8c8e87d574 P 1fa564a3d6e3430eb7d536986cd8023a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:32:41.016031 18719 raft_consensus.cc:2258] T 2dce3e41fef049ec86dd8a8c8e87d574 P 91dec12145b845f78a1a42b0014cd277 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:32:41.016081 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 18238
I20260808 18:32:41.016312 18718 raft_consensus.cc:2258] T 2dce3e41fef049ec86dd8a8c8e87d574 P 1fa564a3d6e3430eb7d536986cd8023a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:32:41.016748 18720 tablet_replica.cc:330] T 2dce3e41fef049ec86dd8a8c8e87d574 P 4e33d84aa7964e04836d0c9cca43534c: stopping tablet replica
I20260808 18:32:41.017052 18720 raft_consensus.cc:2229] T 2dce3e41fef049ec86dd8a8c8e87d574 P 4e33d84aa7964e04836d0c9cca43534c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:32:41.017371 18720 raft_consensus.cc:2258] T 2dce3e41fef049ec86dd8a8c8e87d574 P 4e33d84aa7964e04836d0c9cca43534c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:32:41.019198 18718 ts_tablet_manager.cc:1905] T 2dce3e41fef049ec86dd8a8c8e87d574 P 1fa564a3d6e3430eb7d536986cd8023a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:32:41.022076 18718 ts_tablet_manager.cc:1918] T 2dce3e41fef049ec86dd8a8c8e87d574 P 1fa564a3d6e3430eb7d536986cd8023a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:32:41.022183 18718 log.cc:1196] T 2dce3e41fef049ec86dd8a8c8e87d574 P 1fa564a3d6e3430eb7d536986cd8023a: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/2dce3e41fef049ec86dd8a8c8e87d574
I20260808 18:32:41.022555 18718 ts_tablet_manager.cc:1939] T 2dce3e41fef049ec86dd8a8c8e87d574 P 1fa564a3d6e3430eb7d536986cd8023a: Deleting consensus metadata
I20260808 18:32:41.022576 18720 ts_tablet_manager.cc:1905] T 2dce3e41fef049ec86dd8a8c8e87d574 P 4e33d84aa7964e04836d0c9cca43534c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:32:41.023893 18158 catalog_manager.cc:4841] TS 1fa564a3d6e3430eb7d536986cd8023a (127.29.78.131:37287): tablet 2dce3e41fef049ec86dd8a8c8e87d574 (table test-table [id=2a04d69412ee4d5ba0c3bb70edb90905]) successfully deleted
I20260808 18:32:41.024752 18720 ts_tablet_manager.cc:1918] T 2dce3e41fef049ec86dd8a8c8e87d574 P 4e33d84aa7964e04836d0c9cca43534c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:32:41.024868 18720 log.cc:1196] T 2dce3e41fef049ec86dd8a8c8e87d574 P 4e33d84aa7964e04836d0c9cca43534c: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/2dce3e41fef049ec86dd8a8c8e87d574
I20260808 18:32:41.025206 18720 ts_tablet_manager.cc:1939] T 2dce3e41fef049ec86dd8a8c8e87d574 P 4e33d84aa7964e04836d0c9cca43534c: Deleting consensus metadata
W20260808 18:32:41.025399 18161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:35557 (error 108)
I20260808 18:32:41.025637 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 18379
I20260808 18:32:41.026536 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:41.026144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:35557 (local address 127.0.0.1:53102)
0808 18:32:41.026297 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:41.026392 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:35557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 18:32:41.026721 18161 catalog_manager.cc:4584] TS 91dec12145b845f78a1a42b0014cd277 (127.29.78.129:35557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2dce3e41fef049ec86dd8a8c8e87d574: Network error: Client connection negotiation failed: client connection to 127.29.78.129:35557: connect: Connection refused (error 111)
I20260808 18:32:41.032200 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 18529
I20260808 18:32:41.033814 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:41.032299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:35455 (local address 127.0.0.1:54776)
0808 18:32:41.033622 (+ 1323us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:41.033698 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.130:35455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1211}
I20260808 18:32:41.040395 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 18142
2026-08-08T18:32:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1287 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:32:41 dist-test-slave-gp6n krb5kdc[18724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:32:41 dist-test-slave-gp6n krb5kdc[18724](info): set up 2 sockets
Aug 08 18:32:41 dist-test-slave-gp6n krb5kdc[18724](info): commencing operation
krb5kdc: starting...
W20260808 18:32:42.155473 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:32:42 dist-test-slave-gp6n krb5kdc[18724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:32:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:32:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:32:42.289402 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:46101
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:46101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:42.421466 18740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:42.421756 18740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:42.421828 18740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:42.425830 18740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:32:42.425918 18740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:42.425956 18740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:42.426008 18740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:32:42.426043 18740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:32:42.431267 18740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:46101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:46101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:42.432330 18740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:42.433363 18740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:42.442015 18746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:42.442015 18745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:42.442011 18748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:42.442452 18740 server_base.cc:1034] running on GCE node
I20260808 18:32:42.442997 18740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:42.444098 18740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:42.445297 18740 hybrid_clock.cc:648] HybridClock initialized: now 1786213962445280 us; error 29 us; skew 500 ppm
Aug 08 18:32:42 dist-test-slave-gp6n krb5kdc[18724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213962, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:42.448215 18740 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:32:42.449150 18740 webserver.cc:460] Webserver started at http://127.29.78.190:43261/ using document root <none> and password file <none>
I20260808 18:32:42.449725 18740 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:42.449827 18740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:42.450062 18740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:42.452136 18740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "9aa7b4dbee3d4549a4ccb5bb07e288fd"
format_stamp: "Formatted at 2026-08-08 18:32:42 on dist-test-slave-gp6n"
server_key: "8e0cc45b4f2a2c07aa8da4ae400e1a84"
server_key_iv: "e126ef8d1ea82f946e099dc1027a7f44"
server_key_version: "encryptionkey@0"
I20260808 18:32:42.452710 18740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "9aa7b4dbee3d4549a4ccb5bb07e288fd"
format_stamp: "Formatted at 2026-08-08 18:32:42 on dist-test-slave-gp6n"
server_key: "8e0cc45b4f2a2c07aa8da4ae400e1a84"
server_key_iv: "e126ef8d1ea82f946e099dc1027a7f44"
server_key_version: "encryptionkey@0"
I20260808 18:32:42.456467 18740 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 18:32:42.459095 18755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:42.460242 18740 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260808 18:32:42.460384 18740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "9aa7b4dbee3d4549a4ccb5bb07e288fd"
format_stamp: "Formatted at 2026-08-08 18:32:42 on dist-test-slave-gp6n"
server_key: "8e0cc45b4f2a2c07aa8da4ae400e1a84"
server_key_iv: "e126ef8d1ea82f946e099dc1027a7f44"
server_key_version: "encryptionkey@0"
I20260808 18:32:42.460530 18740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:42.471560 18740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:42.474466 18740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:42.474665 18740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:42.481781 18740 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:46101
I20260808 18:32:42.481808 18812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:46101 every 8 connection(s)
I20260808 18:32:42.482722 18740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:32:42.485235 18813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:42.488884 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 18740
I20260808 18:32:42.489022 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:32:42.489392 30010 external_mini_cluster.cc:1428] Setting key a426ee716500062d80a78e846a2430ae
I20260808 18:32:42.491355 18813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd: Bootstrap starting.
I20260808 18:32:42.493480 18813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:42.494205 18813 log.cc:824] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:42.496346 18813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd: No bootstrap required, opened a new log
I20260808 18:32:42.500007 18813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa7b4dbee3d4549a4ccb5bb07e288fd" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46101 } }
I20260808 18:32:42.500252 18813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:42.500335 18813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa7b4dbee3d4549a4ccb5bb07e288fd, State: Initialized, Role: FOLLOWER
I20260808 18:32:42.500763 18813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa7b4dbee3d4549a4ccb5bb07e288fd" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46101 } }
I20260808 18:32:42.500921 18813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:32:42.501008 18813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:32:42.501127 18813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:42.502417 18813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa7b4dbee3d4549a4ccb5bb07e288fd" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46101 } }
I20260808 18:32:42.502756 18813 leader_election.cc:302] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9aa7b4dbee3d4549a4ccb5bb07e288fd; no voters:
Aug 08 18:32:42 dist-test-slave-gp6n krb5kdc[18724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:32:42.503067 18813 leader_election.cc:288] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:32:42.503189 18819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:32:42.503418 18819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd [term 1 LEADER]: Becoming Leader. State: Replica: 9aa7b4dbee3d4549a4ccb5bb07e288fd, State: Running, Role: LEADER
I20260808 18:32:42.503818 18819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa7b4dbee3d4549a4ccb5bb07e288fd" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46101 } }
I20260808 18:32:42.504419 18813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:32:42.505622 18821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9aa7b4dbee3d4549a4ccb5bb07e288fd. Latest consensus state: current_term: 1 leader_uuid: "9aa7b4dbee3d4549a4ccb5bb07e288fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa7b4dbee3d4549a4ccb5bb07e288fd" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46101 } } }
I20260808 18:32:42.505748 18821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:42.506103 18820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9aa7b4dbee3d4549a4ccb5bb07e288fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa7b4dbee3d4549a4ccb5bb07e288fd" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46101 } } }
I20260808 18:32:42.506227 18820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:42.506618 18828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:32:42.509609 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:42.490835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51454 (local address 127.29.78.190:46101)
0808 18:32:42.491254 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:42.491269 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:42.491302 (+ 33us) server_negotiation.cc:407] Connection header received
0808 18:32:42.491985 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:42.492001 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:42.492315 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:42.492652 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:42.493479 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:42.494652 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:42.495880 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:42.496353 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:42.496537 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:42.503797 (+ 7260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:42.503804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:42.503821 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:42.503858 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:42.506036 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:42.506610 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:42.506615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:42.506621 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:42.506756 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:42.507340 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:42.507343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:42.507346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:42.507869 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:42.508078 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:42.508346 (+ 268us) server_negotiation.cc:299] Negotiation successful
0808 18:32:42.508656 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":120,"threads_started":1}
I20260808 18:32:42.511310 18828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:32:42.516661 18828 catalog_manager.cc:1347] Generated new cluster ID: 03f95ff36e0a4cfe9cf39d6476f97a71
I20260808 18:32:42.516726 18828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:32:42.538441 18828 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:32:42.539386 18828 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:32:42.543291 18828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd: Generated new TSK 0
I20260808 18:32:42.543980 18828 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260808 18:32:42.581250 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:46101
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:42.694125 18843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:42.694363 18843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:42.694465 18843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:42.698360 18843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:42.698472 18843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:42.698577 18843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:32:42.703763 18843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:46101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:42.704807 18843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:42.705956 18843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:42.715426 18852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:42.715463 18849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:42.715608 18843 server_base.cc:1034] running on GCE node
W20260808 18:32:42.715464 18850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:42.716037 18843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:42.716691 18843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:42.717887 18843 hybrid_clock.cc:648] HybridClock initialized: now 1786213962717876 us; error 25 us; skew 500 ppm
Aug 08 18:32:42 dist-test-slave-gp6n krb5kdc[18724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213962, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:42.720631 18843 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260808 18:32:42.721432 18843 webserver.cc:460] Webserver started at http://127.29.78.129:43403/ using document root <none> and password file <none>
I20260808 18:32:42.721977 18843 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:42.722036 18843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:42.722261 18843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:42.724272 18843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "2a07843b43b54305950183a54d78beba"
format_stamp: "Formatted at 2026-08-08 18:32:42 on dist-test-slave-gp6n"
server_key: "f966772545a592c42c7e04fd37859855"
server_key_iv: "51c645b98fbb51aed1dd41b4c39ec1c1"
server_key_version: "encryptionkey@0"
I20260808 18:32:42.724831 18843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "2a07843b43b54305950183a54d78beba"
format_stamp: "Formatted at 2026-08-08 18:32:42 on dist-test-slave-gp6n"
server_key: "f966772545a592c42c7e04fd37859855"
server_key_iv: "51c645b98fbb51aed1dd41b4c39ec1c1"
server_key_version: "encryptionkey@0"
I20260808 18:32:42.728554 18843 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 18:32:42.731002 18859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:42.732405 18843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:32:42.732564 18843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "2a07843b43b54305950183a54d78beba"
format_stamp: "Formatted at 2026-08-08 18:32:42 on dist-test-slave-gp6n"
server_key: "f966772545a592c42c7e04fd37859855"
server_key_iv: "51c645b98fbb51aed1dd41b4c39ec1c1"
server_key_version: "encryptionkey@0"
I20260808 18:32:42.732692 18843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:42.767119 18843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:42.770690 18843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:42.770921 18843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:42.771582 18843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:32:42.772490 18843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:32:42.772567 18843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:42.772647 18843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:32:42.772706 18843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:42.782315 18843 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:33695
I20260808 18:32:42.782348 18982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:33695 every 8 connection(s)
I20260808 18:32:42.783290 18843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:32:42.788192 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 18843
I20260808 18:32:42.788348 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:32:42.788718 30010 external_mini_cluster.cc:1428] Setting key d34c5d0f6f8fb8ee06542ed71dafb27f
Aug 08 18:32:42 dist-test-slave-gp6n krb5kdc[18724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213962, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260808 18:32:42.799544 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:42.785310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:41241 (local address 127.29.78.190:46101)
0808 18:32:42.785511 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:42.785515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:42.786329 (+ 814us) server_negotiation.cc:407] Connection header received
0808 18:32:42.787967 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:42.787970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:42.788030 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:42.788117 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:42.789721 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:42.790452 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:42.792239 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:42.792497 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:42.792631 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:42.795504 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:42.795508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:42.795510 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:42.795536 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:42.797186 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:42.797834 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:42.797837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:42.797838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:42.797921 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:42.798299 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:42.798304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:42.798306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:42.798572 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:42.798690 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:42.799215 (+ 525us) server_negotiation.cc:299] Negotiation successful
0808 18:32:42.799357 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":97}
I20260808 18:32:42.800398 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:42.785687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46101 (local address 127.29.78.129:41241)
0808 18:32:42.786161 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:42.786207 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:42.787693 (+ 1486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:42.788367 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:42.788378 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:42.788916 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:32:42.789536 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:42.789582 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:42.790589 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:42.790593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:42.792093 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:42.792109 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:42.792682 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:42.792686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:42.792869 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:42.793486 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:32:42.793501 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:32:42.795300 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:32:42.797351 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:32:42.797357 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:32:42.797372 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:32:42.797694 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:32:42.798025 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:32:42.798029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:32:42.798032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:32:42.798189 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:32:42.798690 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:32:42.798699 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:32:42.798960 (+ 261us) client_negotiation.cc:769] Sending connection context
0808 18:32:42.799168 (+ 208us) client_negotiation.cc:240] Negotiation successful
0808 18:32:42.799526 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":108,"threads_started":1}
I20260808 18:32:42.801728 18983 heartbeater.cc:347] Connected to a master server at 127.29.78.190:46101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
I20260808 18:32:42.802006 18983 heartbeater.cc:464] Registering TS with master...
I20260808 18:32:42.802620 18983 heartbeater.cc:511] Master 127.29.78.190:46101 requested a full tablet report, sending...
I20260808 18:32:42.804409 18773 ts_manager.cc:194] Registered new tserver with Master: 2a07843b43b54305950183a54d78beba (127.29.78.129:33695)
I20260808 18:32:42.805781 18773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:41241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260808 18:32:42.819876 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:46101
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:42.943078 18989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:42.943368 18989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:42.943413 18989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:42.947561 18989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:42.947619 18989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:42.947719 18989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:32:42.952965 18989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:46101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:42.954021 18989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:42.955173 18989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:42.964524 18995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:42.964536 18996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:42.964664 18998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:42.964994 18989 server_base.cc:1034] running on GCE node
I20260808 18:32:42.965394 18989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:42.966059 18989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:42.967236 18989 hybrid_clock.cc:648] HybridClock initialized: now 1786213962967218 us; error 34 us; skew 500 ppm
Aug 08 18:32:42 dist-test-slave-gp6n krb5kdc[18724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213962, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:42.969950 18989 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260808 18:32:42.970889 18989 webserver.cc:460] Webserver started at http://127.29.78.130:44685/ using document root <none> and password file <none>
I20260808 18:32:42.971444 18989 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:42.971535 18989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:42.971769 18989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:42.973817 18989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "deedfe1e7f2d44d0a4b031e0cd985aaa"
format_stamp: "Formatted at 2026-08-08 18:32:42 on dist-test-slave-gp6n"
server_key: "ba711299e514b4d8fd782d2adbf5259c"
server_key_iv: "cea67e1e0ae0cdf0e8c4d79d7d992f8e"
server_key_version: "encryptionkey@0"
I20260808 18:32:42.974464 18989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "deedfe1e7f2d44d0a4b031e0cd985aaa"
format_stamp: "Formatted at 2026-08-08 18:32:42 on dist-test-slave-gp6n"
server_key: "ba711299e514b4d8fd782d2adbf5259c"
server_key_iv: "cea67e1e0ae0cdf0e8c4d79d7d992f8e"
server_key_version: "encryptionkey@0"
I20260808 18:32:42.978097 18989 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:32:42.980650 19007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:42.981844 18989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:32:42.981990 18989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "deedfe1e7f2d44d0a4b031e0cd985aaa"
format_stamp: "Formatted at 2026-08-08 18:32:42 on dist-test-slave-gp6n"
server_key: "ba711299e514b4d8fd782d2adbf5259c"
server_key_iv: "cea67e1e0ae0cdf0e8c4d79d7d992f8e"
server_key_version: "encryptionkey@0"
I20260808 18:32:42.982101 18989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:43.012861 18989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:43.015780 18989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:43.015990 18989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:43.016587 18989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:32:43.017441 18989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:32:43.017511 18989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:43.017581 18989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:32:43.017635 18989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:43.026378 18989 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:43335
I20260808 18:32:43.026453 19141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:43335 every 8 connection(s)
I20260808 18:32:43.027284 18989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
Aug 08 18:32:43 dist-test-slave-gp6n krb5kdc[18724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213962, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:32:43.037120 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 18989
I20260808 18:32:43.037235 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:32:43.037613 30010 external_mini_cluster.cc:1428] Setting key 905b38b3cf3e9ef2d7520700f1df0fb6
I20260808 18:32:43.041222 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:43.029025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:55711 (local address 127.29.78.190:46101)
0808 18:32:43.029160 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:43.029164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:43.029963 (+ 799us) server_negotiation.cc:407] Connection header received
0808 18:32:43.031456 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:43.031459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:43.031521 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:43.031635 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:43.032796 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.033341 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.034516 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.034745 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.034858 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:43.037455 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:43.037461 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:43.037468 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:43.037501 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:43.039199 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:43.039710 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:43.039715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:43.039716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:43.039801 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:43.040150 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:43.040155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:43.040157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:43.040341 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:43.040452 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:43.040937 (+ 485us) server_negotiation.cc:299] Negotiation successful
0808 18:32:43.041065 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 18:32:43.042044 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:43.029397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46101 (local address 127.29.78.130:55711)
0808 18:32:43.029818 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:43.029860 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:43.031234 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:43.031810 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:43.031817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:43.032172 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:32:43.032645 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:43.032659 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.033472 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.033481 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:43.034361 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:43.034370 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.034899 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.034903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:43.035070 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:43.035676 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:32:43.035691 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:32:43.037248 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:32:43.039319 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:32:43.039325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:32:43.039339 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:32:43.039605 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:32:43.039892 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:32:43.039897 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:32:43.039899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:32:43.040049 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:32:43.040449 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:32:43.040457 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:32:43.040702 (+ 245us) client_negotiation.cc:769] Sending connection context
0808 18:32:43.040892 (+ 190us) client_negotiation.cc:240] Negotiation successful
0808 18:32:43.041229 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":106,"threads_started":1}
I20260808 18:32:43.043375 19142 heartbeater.cc:347] Connected to a master server at 127.29.78.190:46101
I20260808 18:32:43.043615 19142 heartbeater.cc:464] Registering TS with master...
I20260808 18:32:43.044242 19142 heartbeater.cc:511] Master 127.29.78.190:46101 requested a full tablet report, sending...
I20260808 18:32:43.045465 18773 ts_manager.cc:194] Registered new tserver with Master: deedfe1e7f2d44d0a4b031e0cd985aaa (127.29.78.130:43335)
I20260808 18:32:43.046350 18773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:55711
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260808 18:32:43.069832 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:46101
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:43.185002 19149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:43.185315 19149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:43.185405 19149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:43.189473 19149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:43.189560 19149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:43.189689 19149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:32:43.194996 19149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:46101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:43.196027 19149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:43.197033 19149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:43.206780 19157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:43.206780 19154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:43.206925 19149 server_base.cc:1034] running on GCE node
W20260808 18:32:43.206818 19155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:43.207369 19149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:43.208001 19149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:43.209164 19149 hybrid_clock.cc:648] HybridClock initialized: now 1786213963209144 us; error 32 us; skew 500 ppm
Aug 08 18:32:43 dist-test-slave-gp6n krb5kdc[18724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213963, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:43.212093 19149 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260808 18:32:43.212988 19149 webserver.cc:460] Webserver started at http://127.29.78.131:36741/ using document root <none> and password file <none>
I20260808 18:32:43.213554 19149 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:43.213647 19149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:43.213879 19149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:43.216029 19149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "4841063d75ee41f9870f4501ffdf4d75"
format_stamp: "Formatted at 2026-08-08 18:32:43 on dist-test-slave-gp6n"
server_key: "6d2dc8947efed2bf17c3fa876dfc56f8"
server_key_iv: "118cba92472be683f432f52de4b204f1"
server_key_version: "encryptionkey@0"
I20260808 18:32:43.216662 19149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "4841063d75ee41f9870f4501ffdf4d75"
format_stamp: "Formatted at 2026-08-08 18:32:43 on dist-test-slave-gp6n"
server_key: "6d2dc8947efed2bf17c3fa876dfc56f8"
server_key_iv: "118cba92472be683f432f52de4b204f1"
server_key_version: "encryptionkey@0"
I20260808 18:32:43.220412 19149 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 18:32:43.222826 19165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:43.224040 19149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:32:43.224186 19149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "4841063d75ee41f9870f4501ffdf4d75"
format_stamp: "Formatted at 2026-08-08 18:32:43 on dist-test-slave-gp6n"
server_key: "6d2dc8947efed2bf17c3fa876dfc56f8"
server_key_iv: "118cba92472be683f432f52de4b204f1"
server_key_version: "encryptionkey@0"
I20260808 18:32:43.224299 19149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:43.237931 19149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:43.240797 19149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:43.241001 19149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:43.241583 19149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:32:43.242425 19149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:32:43.242494 19149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:43.242569 19149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:32:43.242622 19149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:43.251286 19149 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:42785
I20260808 18:32:43.251313 19290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:42785 every 8 connection(s)
I20260808 18:32:43.252148 19149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:32:43.256773 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 19149
I20260808 18:32:43.256902 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:32:43.257237 30010 external_mini_cluster.cc:1428] Setting key 4707e2be54d4f8953de9d0ad47d67cd2
Aug 08 18:32:43 dist-test-slave-gp6n krb5kdc[18724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213963, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:32:43.266110 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:43.253881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:54577 (local address 127.29.78.190:46101)
0808 18:32:43.254039 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:43.254044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:43.254927 (+ 883us) server_negotiation.cc:407] Connection header received
0808 18:32:43.256149 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:43.256152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:43.256198 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:43.256273 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:43.257794 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.258566 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.259870 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.260088 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.260171 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:43.262582 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:43.262585 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:43.262587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:43.262612 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:43.264136 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:43.264751 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:43.264754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:43.264755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:43.264833 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:43.265146 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:43.265148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:43.265150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:43.265323 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:43.265403 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:43.265888 (+ 485us) server_negotiation.cc:299] Negotiation successful
0808 18:32:43.265989 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 18:32:43.267005 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:43.254276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46101 (local address 127.29.78.131:54577)
0808 18:32:43.254768 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:43.254809 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:43.255959 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:43.256436 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:43.256442 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:43.256983 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:32:43.257631 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:43.257647 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.258694 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.258697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:43.259756 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:43.259765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.260216 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.260219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:43.260360 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:43.260872 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:32:43.260888 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:32:43.262362 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:32:43.264239 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:32:43.264244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:32:43.264257 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:32:43.264661 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:32:43.264920 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:32:43.264923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:32:43.264925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:32:43.265080 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:32:43.265453 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:32:43.265458 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:32:43.265683 (+ 225us) client_negotiation.cc:769] Sending connection context
0808 18:32:43.265841 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 18:32:43.266093 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":182,"threads_started":1}
I20260808 18:32:43.268118 19291 heartbeater.cc:347] Connected to a master server at 127.29.78.190:46101
I20260808 18:32:43.268378 19291 heartbeater.cc:464] Registering TS with master...
I20260808 18:32:43.268867 19291 heartbeater.cc:511] Master 127.29.78.190:46101 requested a full tablet report, sending...
I20260808 18:32:43.269935 18773 ts_manager.cc:194] Registered new tserver with Master: 4841063d75ee41f9870f4501ffdf4d75 (127.29.78.131:42785)
I20260808 18:32:43.270674 18773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:54577
I20260808 18:32:43.271128 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:32:43.280315 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:43.273141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51456 (local address 127.29.78.190:46101)
0808 18:32:43.273261 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:43.273264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:43.273476 (+ 212us) server_negotiation.cc:407] Connection header received
0808 18:32:43.273565 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:43.273568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:43.273602 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:43.273661 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:43.274304 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.274897 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.275716 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.275953 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.276054 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:43.277446 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:43.277450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:43.277452 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:43.277481 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:43.278887 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:43.279291 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:43.279293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:43.279294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:43.279366 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:43.279605 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:43.279608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:43.279609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:43.279811 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:43.279927 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:43.280123 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 18:32:43.280212 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:32:43.283217 18773 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:32:43.286727 18773 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:32:43.301972 19302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:43.295557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:43335 (local address 127.0.0.1:37364)
0808 18:32:43.296753 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:43.296791 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:43.296923 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:43.297560 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:43.297563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:43.297580 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:43.297700 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:43.297719 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.299307 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.299310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:43.300563 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:43.300571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.301532 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.301536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:43.301648 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:43.301669 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 18:32:43.301711 (+ 42us) client_negotiation.cc:240] Negotiation successful
0808 18:32:43.301801 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":100,"spinlock_wait_cycles":677376,"thread_start_us":115,"threads_started":1}
I20260808 18:32:43.301982 19307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:43.296670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58166 (local address 127.29.78.131:42785)
0808 18:32:43.297056 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:43.297067 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:43.297090 (+ 23us) server_negotiation.cc:407] Connection header received
0808 18:32:43.297171 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:43.297176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:43.297369 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:43.297528 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:43.297822 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.299091 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.300360 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.301223 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.301340 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:43.301515 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:43.301701 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 18:32:43.301803 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":141,"threads_started":1}
I20260808 18:32:43.302178 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:43.295661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57770 (local address 127.29.78.129:33695)
0808 18:32:43.296068 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:43.296089 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:43.297190 (+ 1101us) server_negotiation.cc:407] Connection header received
0808 18:32:43.297264 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:43.297269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:43.297412 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:43.297550 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:43.297938 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.298829 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.300529 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.301333 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.301424 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:43.301533 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:43.301922 (+ 389us) server_negotiation.cc:299] Negotiation successful
0808 18:32:43.302030 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":79,"threads_started":1}
I20260808 18:32:43.302664 19305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:43.296290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:42785 (local address 127.0.0.1:58166)
0808 18:32:43.296755 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:43.296797 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:43.296921 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:43.297526 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:43.297532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:43.297554 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:43.297695 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:43.297702 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.299253 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.299257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:43.300223 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:43.300232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.301362 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.301366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:43.301492 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:43.301543 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 18:32:43.301654 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 18:32:43.302537 (+ 883us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":105,"thread_start_us":92,"threads_started":1}
I20260808 18:32:43.302824 19303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:43.295543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:33695 (local address 127.0.0.1:57770)
0808 18:32:43.297079 (+ 1536us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:43.297092 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:43.297176 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:43.297695 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:43.297698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:43.297712 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:43.297826 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:43.297831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.298971 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.298974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:43.300401 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:43.300413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.301594 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.301598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:43.301756 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:43.301782 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 18:32:43.302539 (+ 757us) client_negotiation.cc:240] Negotiation successful
0808 18:32:43.302623 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":53,"thread_start_us":79,"threads_started":1}
I20260808 18:32:43.303103 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:43.296328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37364 (local address 127.29.78.130:43335)
0808 18:32:43.296654 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:43.296664 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:43.297091 (+ 427us) server_negotiation.cc:407] Connection header received
0808 18:32:43.297189 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:43.297194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:43.297386 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:43.297885 (+ 499us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:43.298026 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.299174 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.300673 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.301383 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.302348 (+ 965us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:43.302544 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:43.302598 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 18:32:43.302694 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":115,"threads_started":1}
I20260808 18:32:43.305625 19061 tablet_service.cc:1467] Processing CreateTablet for tablet bf8c9df306b84542b06f30cf4d059d05 (DEFAULT_TABLE table=test-table [id=081503d76850479ca04b177cd14dbdb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:32:43.305624 18913 tablet_service.cc:1467] Processing CreateTablet for tablet bf8c9df306b84542b06f30cf4d059d05 (DEFAULT_TABLE table=test-table [id=081503d76850479ca04b177cd14dbdb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:32:43.305565 19216 tablet_service.cc:1467] Processing CreateTablet for tablet bf8c9df306b84542b06f30cf4d059d05 (DEFAULT_TABLE table=test-table [id=081503d76850479ca04b177cd14dbdb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:32:43.306762 19216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf8c9df306b84542b06f30cf4d059d05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:43.306766 18913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf8c9df306b84542b06f30cf4d059d05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:43.306764 19061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf8c9df306b84542b06f30cf4d059d05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:43.312718 19308 tablet_bootstrap.cc:492] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75: Bootstrap starting.
I20260808 18:32:43.314105 19309 tablet_bootstrap.cc:492] T bf8c9df306b84542b06f30cf4d059d05 P 2a07843b43b54305950183a54d78beba: Bootstrap starting.
I20260808 18:32:43.314460 19310 tablet_bootstrap.cc:492] T bf8c9df306b84542b06f30cf4d059d05 P deedfe1e7f2d44d0a4b031e0cd985aaa: Bootstrap starting.
I20260808 18:32:43.315127 19308 tablet_bootstrap.cc:654] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:43.316252 19309 tablet_bootstrap.cc:654] T bf8c9df306b84542b06f30cf4d059d05 P 2a07843b43b54305950183a54d78beba: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:43.316296 19310 tablet_bootstrap.cc:654] T bf8c9df306b84542b06f30cf4d059d05 P deedfe1e7f2d44d0a4b031e0cd985aaa: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:43.316370 19308 log.cc:824] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:43.317149 19309 log.cc:824] T bf8c9df306b84542b06f30cf4d059d05 P 2a07843b43b54305950183a54d78beba: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:43.317454 19310 log.cc:824] T bf8c9df306b84542b06f30cf4d059d05 P deedfe1e7f2d44d0a4b031e0cd985aaa: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:43.318502 19308 tablet_bootstrap.cc:492] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75: No bootstrap required, opened a new log
I20260808 18:32:43.318697 19308 ts_tablet_manager.cc:1397] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260808 18:32:43.319154 19309 tablet_bootstrap.cc:492] T bf8c9df306b84542b06f30cf4d059d05 P 2a07843b43b54305950183a54d78beba: No bootstrap required, opened a new log
I20260808 18:32:43.319355 19309 ts_tablet_manager.cc:1397] T bf8c9df306b84542b06f30cf4d059d05 P 2a07843b43b54305950183a54d78beba: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 18:32:43.319429 19310 tablet_bootstrap.cc:492] T bf8c9df306b84542b06f30cf4d059d05 P deedfe1e7f2d44d0a4b031e0cd985aaa: No bootstrap required, opened a new log
I20260808 18:32:43.319609 19310 ts_tablet_manager.cc:1397] T bf8c9df306b84542b06f30cf4d059d05 P deedfe1e7f2d44d0a4b031e0cd985aaa: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 18:32:43.322618 19308 raft_consensus.cc:348] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07843b43b54305950183a54d78beba" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 33695 } } peers { permanent_uuid: "deedfe1e7f2d44d0a4b031e0cd985aaa" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43335 } } peers { permanent_uuid: "4841063d75ee41f9870f4501ffdf4d75" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42785 } }
I20260808 18:32:43.322602 19309 raft_consensus.cc:348] T bf8c9df306b84542b06f30cf4d059d05 P 2a07843b43b54305950183a54d78beba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07843b43b54305950183a54d78beba" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 33695 } } peers { permanent_uuid: "deedfe1e7f2d44d0a4b031e0cd985aaa" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43335 } } peers { permanent_uuid: "4841063d75ee41f9870f4501ffdf4d75" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42785 } }
I20260808 18:32:43.322891 19308 raft_consensus.cc:374] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:43.322897 19309 raft_consensus.cc:374] T bf8c9df306b84542b06f30cf4d059d05 P 2a07843b43b54305950183a54d78beba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:43.322950 19308 raft_consensus.cc:729] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4841063d75ee41f9870f4501ffdf4d75, State: Initialized, Role: FOLLOWER
I20260808 18:32:43.322955 19309 raft_consensus.cc:729] T bf8c9df306b84542b06f30cf4d059d05 P 2a07843b43b54305950183a54d78beba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a07843b43b54305950183a54d78beba, State: Initialized, Role: FOLLOWER
I20260808 18:32:43.323333 19310 raft_consensus.cc:348] T bf8c9df306b84542b06f30cf4d059d05 P deedfe1e7f2d44d0a4b031e0cd985aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07843b43b54305950183a54d78beba" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 33695 } } peers { permanent_uuid: "deedfe1e7f2d44d0a4b031e0cd985aaa" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43335 } } peers { permanent_uuid: "4841063d75ee41f9870f4501ffdf4d75" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42785 } }
I20260808 18:32:43.323392 19309 consensus_queue.cc:260] T bf8c9df306b84542b06f30cf4d059d05 P 2a07843b43b54305950183a54d78beba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07843b43b54305950183a54d78beba" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 33695 } } peers { permanent_uuid: "deedfe1e7f2d44d0a4b031e0cd985aaa" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43335 } } peers { permanent_uuid: "4841063d75ee41f9870f4501ffdf4d75" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42785 } }
I20260808 18:32:43.323563 19310 raft_consensus.cc:374] T bf8c9df306b84542b06f30cf4d059d05 P deedfe1e7f2d44d0a4b031e0cd985aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:43.323477 19308 consensus_queue.cc:260] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07843b43b54305950183a54d78beba" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 33695 } } peers { permanent_uuid: "deedfe1e7f2d44d0a4b031e0cd985aaa" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43335 } } peers { permanent_uuid: "4841063d75ee41f9870f4501ffdf4d75" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42785 } }
I20260808 18:32:43.323608 19310 raft_consensus.cc:729] T bf8c9df306b84542b06f30cf4d059d05 P deedfe1e7f2d44d0a4b031e0cd985aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deedfe1e7f2d44d0a4b031e0cd985aaa, State: Initialized, Role: FOLLOWER
I20260808 18:32:43.324031 19310 consensus_queue.cc:260] T bf8c9df306b84542b06f30cf4d059d05 P deedfe1e7f2d44d0a4b031e0cd985aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07843b43b54305950183a54d78beba" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 33695 } } peers { permanent_uuid: "deedfe1e7f2d44d0a4b031e0cd985aaa" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43335 } } peers { permanent_uuid: "4841063d75ee41f9870f4501ffdf4d75" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42785 } }
I20260808 18:32:43.324152 19291 heartbeater.cc:503] Master 127.29.78.190:46101 was elected leader, sending a full tablet report...
I20260808 18:32:43.324441 18983 heartbeater.cc:503] Master 127.29.78.190:46101 was elected leader, sending a full tablet report...
I20260808 18:32:43.324512 19309 ts_tablet_manager.cc:1428] T bf8c9df306b84542b06f30cf4d059d05 P 2a07843b43b54305950183a54d78beba: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 18:32:43.324888 19310 ts_tablet_manager.cc:1428] T bf8c9df306b84542b06f30cf4d059d05 P deedfe1e7f2d44d0a4b031e0cd985aaa: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 18:32:43.325187 19142 heartbeater.cc:503] Master 127.29.78.190:46101 was elected leader, sending a full tablet report...
I20260808 18:32:43.326303 19308 ts_tablet_manager.cc:1428] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260808 18:32:43.384893 19315 raft_consensus.cc:482] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:32:43.385160 19315 raft_consensus.cc:504] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07843b43b54305950183a54d78beba" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 33695 } } peers { permanent_uuid: "deedfe1e7f2d44d0a4b031e0cd985aaa" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43335 } } peers { permanent_uuid: "4841063d75ee41f9870f4501ffdf4d75" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42785 } }
I20260808 18:32:43.386402 19315 leader_election.cc:288] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a07843b43b54305950183a54d78beba (127.29.78.129:33695), deedfe1e7f2d44d0a4b031e0cd985aaa (127.29.78.130:43335)
I20260808 18:32:43.391113 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:43.386688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:33695 (local address 127.29.78.131:56137)
0808 18:32:43.387301 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:43.387315 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:43.387407 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:43.387663 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:43.387665 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:43.387680 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:43.387773 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:43.387777 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.388675 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.388677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:43.389816 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:43.389823 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.390658 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.390664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:43.390765 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:43.390788 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 18:32:43.390865 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 18:32:43.390930 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":214,"thread_start_us":162,"threads_started":1}
I20260808 18:32:43.391131 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:43.386846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:56137 (local address 127.29.78.129:33695)
0808 18:32:43.387007 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:43.387013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:43.387405 (+ 392us) server_negotiation.cc:407] Connection header received
0808 18:32:43.387489 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:43.387492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:43.387563 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:43.387648 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:43.387885 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.388583 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.390027 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.390533 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.390631 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:43.390695 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:43.390910 (+ 215us) server_negotiation.cc:299] Negotiation successful
0808 18:32:43.391002 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 18:32:43.391223 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:43.386690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:43335 (local address 127.29.78.131:44563)
0808 18:32:43.386813 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:43.386830 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:43.386955 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:43.387259 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:43.387264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:43.387282 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:43.387474 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:43.387484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.388508 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.388511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:43.389794 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:43.389805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.390750 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.390754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:43.390883 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:43.390907 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 18:32:43.390989 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 18:32:43.391060 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 18:32:43.391227 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:43.386856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:44563 (local address 127.29.78.130:43335)
0808 18:32:43.386980 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:43.386985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:43.387006 (+ 21us) server_negotiation.cc:407] Connection header received
0808 18:32:43.387071 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:43.387077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:43.387137 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:43.387239 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:43.387608 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.388393 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.389951 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.390616 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.390739 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:43.390790 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:43.391021 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 18:32:43.391100 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 18:32:43.391860 19086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf8c9df306b84542b06f30cf4d059d05" candidate_uuid: "4841063d75ee41f9870f4501ffdf4d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deedfe1e7f2d44d0a4b031e0cd985aaa" is_pre_election: true
I20260808 18:32:43.391914 18933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf8c9df306b84542b06f30cf4d059d05" candidate_uuid: "4841063d75ee41f9870f4501ffdf4d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a07843b43b54305950183a54d78beba" is_pre_election: true
I20260808 18:32:43.392207 18933 raft_consensus.cc:2454] T bf8c9df306b84542b06f30cf4d059d05 P 2a07843b43b54305950183a54d78beba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4841063d75ee41f9870f4501ffdf4d75 in term 0.
I20260808 18:32:43.392208 19086 raft_consensus.cc:2454] T bf8c9df306b84542b06f30cf4d059d05 P deedfe1e7f2d44d0a4b031e0cd985aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4841063d75ee41f9870f4501ffdf4d75 in term 0.
I20260808 18:32:43.392710 19169 leader_election.cc:302] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a07843b43b54305950183a54d78beba, 4841063d75ee41f9870f4501ffdf4d75; no voters:
I20260808 18:32:43.392988 19315 raft_consensus.cc:2781] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:32:43.393095 19315 raft_consensus.cc:482] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:32:43.393157 19315 raft_consensus.cc:3037] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:43.394196 19315 raft_consensus.cc:504] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07843b43b54305950183a54d78beba" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 33695 } } peers { permanent_uuid: "deedfe1e7f2d44d0a4b031e0cd985aaa" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43335 } } peers { permanent_uuid: "4841063d75ee41f9870f4501ffdf4d75" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42785 } }
I20260808 18:32:43.394584 19315 leader_election.cc:288] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75 [CANDIDATE]: Term 1 election: Requested vote from peers 2a07843b43b54305950183a54d78beba (127.29.78.129:33695), deedfe1e7f2d44d0a4b031e0cd985aaa (127.29.78.130:43335)
I20260808 18:32:43.394912 18933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf8c9df306b84542b06f30cf4d059d05" candidate_uuid: "4841063d75ee41f9870f4501ffdf4d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a07843b43b54305950183a54d78beba"
I20260808 18:32:43.394917 19086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf8c9df306b84542b06f30cf4d059d05" candidate_uuid: "4841063d75ee41f9870f4501ffdf4d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deedfe1e7f2d44d0a4b031e0cd985aaa"
I20260808 18:32:43.395011 18933 raft_consensus.cc:3037] T bf8c9df306b84542b06f30cf4d059d05 P 2a07843b43b54305950183a54d78beba [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:43.395059 19086 raft_consensus.cc:3037] T bf8c9df306b84542b06f30cf4d059d05 P deedfe1e7f2d44d0a4b031e0cd985aaa [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:43.396093 18933 raft_consensus.cc:2454] T bf8c9df306b84542b06f30cf4d059d05 P 2a07843b43b54305950183a54d78beba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4841063d75ee41f9870f4501ffdf4d75 in term 1.
I20260808 18:32:43.396176 19086 raft_consensus.cc:2454] T bf8c9df306b84542b06f30cf4d059d05 P deedfe1e7f2d44d0a4b031e0cd985aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4841063d75ee41f9870f4501ffdf4d75 in term 1.
I20260808 18:32:43.396410 19169 leader_election.cc:302] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a07843b43b54305950183a54d78beba, 4841063d75ee41f9870f4501ffdf4d75; no voters:
I20260808 18:32:43.396620 19315 raft_consensus.cc:2781] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:32:43.396863 19315 raft_consensus.cc:686] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75 [term 1 LEADER]: Becoming Leader. State: Replica: 4841063d75ee41f9870f4501ffdf4d75, State: Running, Role: LEADER
I20260808 18:32:43.397185 19315 consensus_queue.cc:237] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07843b43b54305950183a54d78beba" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 33695 } } peers { permanent_uuid: "deedfe1e7f2d44d0a4b031e0cd985aaa" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43335 } } peers { permanent_uuid: "4841063d75ee41f9870f4501ffdf4d75" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42785 } }
I20260808 18:32:43.400462 18772 catalog_manager.cc:5495] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4841063d75ee41f9870f4501ffdf4d75 (127.29.78.131). New cstate: current_term: 1 leader_uuid: "4841063d75ee41f9870f4501ffdf4d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07843b43b54305950183a54d78beba" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 33695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "deedfe1e7f2d44d0a4b031e0cd985aaa" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4841063d75ee41f9870f4501ffdf4d75" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42785 } health_report { overall_health: HEALTHY } } }
I20260808 18:32:43.421864 19307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:43.418126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58172 (local address 127.29.78.131:42785)
0808 18:32:43.418290 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:43.418297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:43.418347 (+ 50us) server_negotiation.cc:407] Connection header received
0808 18:32:43.418566 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:43.418568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:43.418624 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:43.418738 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:32:43.419010 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.419521 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.420392 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.420609 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.420717 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:43.421042 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:32:43.421450 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:32:43.421583 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:43.421645 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 18:32:43.421701 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 18:32:43.427831 19086 raft_consensus.cc:1261] T bf8c9df306b84542b06f30cf4d059d05 P deedfe1e7f2d44d0a4b031e0cd985aaa [term 1 FOLLOWER]: Refusing update from remote peer 4841063d75ee41f9870f4501ffdf4d75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:32:43.427834 18933 raft_consensus.cc:1261] T bf8c9df306b84542b06f30cf4d059d05 P 2a07843b43b54305950183a54d78beba [term 1 FOLLOWER]: Refusing update from remote peer 4841063d75ee41f9870f4501ffdf4d75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:32:43.428603 19319 consensus_queue.cc:1035] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a07843b43b54305950183a54d78beba" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 33695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:32:43.428823 19315 consensus_queue.cc:1035] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "deedfe1e7f2d44d0a4b031e0cd985aaa" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:32:43.435873 19323 mvcc.cc:205] Tried to move back new op lower bound from 7316332394192543744 to 7316332394077360128. Current Snapshot: MvccSnapshot[applied={T|T < 7316332394192543744}]
I20260808 18:32:43.438862 19324 mvcc.cc:205] Tried to move back new op lower bound from 7316332394192543744 to 7316332394077360128. Current Snapshot: MvccSnapshot[applied={T|T < 7316332394192543744}]
I20260808 18:32:43.444195 18772 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:32:43.444367 18772 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:32:43.446851 18772 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9aa7b4dbee3d4549a4ccb5bb07e288fd: Sending DeleteTablet for 3 replicas of tablet bf8c9df306b84542b06f30cf4d059d05
I20260808 18:32:43.447672 18913 tablet_service.cc:1514] Processing DeleteTablet for tablet bf8c9df306b84542b06f30cf4d059d05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:32:43 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:57770
I20260808 18:32:43.447769 19061 tablet_service.cc:1514] Processing DeleteTablet for tablet bf8c9df306b84542b06f30cf4d059d05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:32:43 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:37364
I20260808 18:32:43.448196 19332 tablet_replica.cc:330] T bf8c9df306b84542b06f30cf4d059d05 P deedfe1e7f2d44d0a4b031e0cd985aaa: stopping tablet replica
I20260808 18:32:43.448181 19216 tablet_service.cc:1514] Processing DeleteTablet for tablet bf8c9df306b84542b06f30cf4d059d05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:32:43 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:58166
I20260808 18:32:43.448720 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 18843
I20260808 18:32:43.451206 19332 raft_consensus.cc:2229] T bf8c9df306b84542b06f30cf4d059d05 P deedfe1e7f2d44d0a4b031e0cd985aaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:32:43.451557 19332 raft_consensus.cc:2258] T bf8c9df306b84542b06f30cf4d059d05 P deedfe1e7f2d44d0a4b031e0cd985aaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:32:43.452005 19333 tablet_replica.cc:330] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75: stopping tablet replica
I20260808 18:32:43.452272 19333 raft_consensus.cc:2229] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:32:43.452591 19333 raft_consensus.cc:2258] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:32:43.455406 19332 ts_tablet_manager.cc:1905] T bf8c9df306b84542b06f30cf4d059d05 P deedfe1e7f2d44d0a4b031e0cd985aaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:32:43.457608 19332 ts_tablet_manager.cc:1918] T bf8c9df306b84542b06f30cf4d059d05 P deedfe1e7f2d44d0a4b031e0cd985aaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:32:43.457720 19332 log.cc:1196] T bf8c9df306b84542b06f30cf4d059d05 P deedfe1e7f2d44d0a4b031e0cd985aaa: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/bf8c9df306b84542b06f30cf4d059d05
I20260808 18:32:43.458094 19332 ts_tablet_manager.cc:1939] T bf8c9df306b84542b06f30cf4d059d05 P deedfe1e7f2d44d0a4b031e0cd985aaa: Deleting consensus metadata
I20260808 18:32:43.458847 19333 ts_tablet_manager.cc:1905] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:32:43.459329 18760 catalog_manager.cc:4841] TS deedfe1e7f2d44d0a4b031e0cd985aaa (127.29.78.130:43335): tablet bf8c9df306b84542b06f30cf4d059d05 (table test-table [id=081503d76850479ca04b177cd14dbdb8]) successfully deleted
W20260808 18:32:43.460944 18759 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:33695 (error 108)
I20260808 18:32:43.461336 19333 ts_tablet_manager.cc:1918] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:32:43.461448 19333 log.cc:1196] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/bf8c9df306b84542b06f30cf4d059d05
I20260808 18:32:43.461807 19333 ts_tablet_manager.cc:1939] T bf8c9df306b84542b06f30cf4d059d05 P 4841063d75ee41f9870f4501ffdf4d75: Deleting consensus metadata
I20260808 18:32:43.461881 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 18989
I20260808 18:32:43.462193 19303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:43.461865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:33695 (local address 127.0.0.1:57774)
0808 18:32:43.462012 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:43.462093 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:33695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260808 18:32:43.462363 18759 catalog_manager.cc:4584] TS 2a07843b43b54305950183a54d78beba (127.29.78.129:33695): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf8c9df306b84542b06f30cf4d059d05: Network error: Client connection negotiation failed: client connection to 127.29.78.129:33695: connect: Connection refused (error 111)
I20260808 18:32:43.463776 18760 catalog_manager.cc:4841] TS 4841063d75ee41f9870f4501ffdf4d75 (127.29.78.131:42785): tablet bf8c9df306b84542b06f30cf4d059d05 (table test-table [id=081503d76850479ca04b177cd14dbdb8]) successfully deleted
I20260808 18:32:43.477408 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 19149
I20260808 18:32:43.484086 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 18740
2026-08-08T18:32:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2442 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-08T18:32:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:32:43Z Disabled control of system clock
I20260808 18:32:43.516429 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:36579
--webserver_interface=127.29.78.190
--webserver_port=0
--builtin_ntp_servers=127.29.78.148:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:36579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 18:32:43.624810 19341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:43.625084 19341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:43.625119 19341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:43.628978 19341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:32:43.629024 19341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:43.629040 19341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:43.629055 19341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:32:43.629070 19341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:32:43.633679 19341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:36579
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:36579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:43.634675 19341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:43.635968 19341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:43.644201 19351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:43.644236 19348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:43.644212 19349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:43.644451 19341 server_base.cc:1034] running on GCE node
I20260808 18:32:43.645067 19341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:43.646126 19341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:43.647330 19341 hybrid_clock.cc:648] HybridClock initialized: now 1786213963647304 us; error 45 us; skew 500 ppm
I20260808 18:32:43.649117 19341 webserver.cc:460] Webserver started at http://127.29.78.190:38577/ using document root <none> and password file <none>
I20260808 18:32:43.649694 19341 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:43.649756 19341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:43.650084 19341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:43.652149 19341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "5e842fcc228149708230f1372bca4d18"
format_stamp: "Formatted at 2026-08-08 18:32:43 on dist-test-slave-gp6n"
server_key: "06915c4689ef3629acd693343795fd25"
server_key_iv: "74e09f4519f6093581de67f6eedd35b5"
server_key_version: "encryptionkey@0"
I20260808 18:32:43.652716 19341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "5e842fcc228149708230f1372bca4d18"
format_stamp: "Formatted at 2026-08-08 18:32:43 on dist-test-slave-gp6n"
server_key: "06915c4689ef3629acd693343795fd25"
server_key_iv: "74e09f4519f6093581de67f6eedd35b5"
server_key_version: "encryptionkey@0"
I20260808 18:32:43.656272 19341 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 18:32:43.658617 19358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:43.659933 19341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:32:43.660081 19341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "5e842fcc228149708230f1372bca4d18"
format_stamp: "Formatted at 2026-08-08 18:32:43 on dist-test-slave-gp6n"
server_key: "06915c4689ef3629acd693343795fd25"
server_key_iv: "74e09f4519f6093581de67f6eedd35b5"
server_key_version: "encryptionkey@0"
I20260808 18:32:43.660192 19341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:43.675206 19341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:43.675974 19341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:43.676185 19341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:43.684141 19341 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:36579
I20260808 18:32:43.684162 19415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:36579 every 8 connection(s)
I20260808 18:32:43.685133 19341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:32:43.688267 19417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:43.691916 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 19341
I20260808 18:32:43.692057 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:32:43.692377 30010 external_mini_cluster.cc:1428] Setting key 2cbb766ca3c51c0386fcb91e1dbfd70f
I20260808 18:32:43.694664 19417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18: Bootstrap starting.
I20260808 18:32:43.696836 19417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:43.697607 19417 log.cc:824] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:43.699584 19417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18: No bootstrap required, opened a new log
I20260808 18:32:43.701640 19420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:43.693862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35236 (local address 127.29.78.190:36579)
0808 18:32:43.694558 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:43.694575 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:43.694610 (+ 35us) server_negotiation.cc:407] Connection header received
0808 18:32:43.695292 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:43.695310 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:43.695697 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:43.696037 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:43.696469 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.697493 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.698436 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.698916 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.699087 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:43.700012 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:43.700017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:43.700029 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:43.700406 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:43.700461 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:43.700480 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:43.700589 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:43.700736 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 18:32:43.700982 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":121,"threads_started":1}
I20260808 18:32:43.703397 19417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e842fcc228149708230f1372bca4d18" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36579 } }
I20260808 18:32:43.703681 19417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:43.703751 19417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e842fcc228149708230f1372bca4d18, State: Initialized, Role: FOLLOWER
I20260808 18:32:43.704172 19417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e842fcc228149708230f1372bca4d18" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36579 } }
I20260808 18:32:43.704299 19417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:32:43.704340 19417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:32:43.704403 19417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:43.705219 19417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e842fcc228149708230f1372bca4d18" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36579 } }
I20260808 18:32:43.705499 19417 leader_election.cc:302] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e842fcc228149708230f1372bca4d18; no voters:
I20260808 18:32:43.705730 19417 leader_election.cc:288] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:32:43.705857 19422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:32:43.706111 19422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18 [term 1 LEADER]: Becoming Leader. State: Replica: 5e842fcc228149708230f1372bca4d18, State: Running, Role: LEADER
I20260808 18:32:43.706498 19422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e842fcc228149708230f1372bca4d18" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36579 } }
I20260808 18:32:43.706954 19417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:32:43.708280 19424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e842fcc228149708230f1372bca4d18. Latest consensus state: current_term: 1 leader_uuid: "5e842fcc228149708230f1372bca4d18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e842fcc228149708230f1372bca4d18" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36579 } } }
I20260808 18:32:43.708402 19423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e842fcc228149708230f1372bca4d18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e842fcc228149708230f1372bca4d18" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36579 } } }
I20260808 18:32:43.708524 19423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18 [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:43.708555 19424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18 [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:43.708896 19434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:32:43.712477 19434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:32:43.717770 19434 catalog_manager.cc:1347] Generated new cluster ID: b4c8aa606c90453bacd16e7ced9317a5
I20260808 18:32:43.717835 19434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:32:43.727737 19434 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:32:43.728575 19434 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:32:43.737567 19434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18: Generated new TSK 0
I20260808 18:32:43.738260 19434 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:32:43.742810 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36579
--builtin_ntp_servers=127.29.78.148:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 18:32:43.856381 19444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:43.856642 19444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:43.856679 19444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:43.860800 19444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:43.860854 19444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:43.860908 19444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260808 18:32:43.866483 19444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:43.867605 19444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:43.869057 19444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:43.878320 19452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:43.878396 19444 server_base.cc:1034] running on GCE node
W20260808 18:32:43.878320 19451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:43.878590 19454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:43.878971 19444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:43.879637 19444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:43.880842 19444 hybrid_clock.cc:648] HybridClock initialized: now 1786213963880811 us; error 27 us; skew 500 ppm
I20260808 18:32:43.882745 19444 webserver.cc:460] Webserver started at http://127.29.78.129:33681/ using document root <none> and password file <none>
I20260808 18:32:43.883360 19444 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:43.883462 19444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:43.883703 19444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:43.885706 19444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-0/data/instance:
uuid: "8068030ce1d8484eb339c8264310627c"
format_stamp: "Formatted at 2026-08-08 18:32:43 on dist-test-slave-gp6n"
server_key: "b204a7f54f8e03ea4b111e96d0d16c05"
server_key_iv: "0bbb0f465f8c302ea7e6c4edc077a006"
server_key_version: "encryptionkey@0"
I20260808 18:32:43.886274 19444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "8068030ce1d8484eb339c8264310627c"
format_stamp: "Formatted at 2026-08-08 18:32:43 on dist-test-slave-gp6n"
server_key: "b204a7f54f8e03ea4b111e96d0d16c05"
server_key_iv: "0bbb0f465f8c302ea7e6c4edc077a006"
server_key_version: "encryptionkey@0"
I20260808 18:32:43.889868 19444 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:32:43.892403 19460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:43.893606 19444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:32:43.893740 19444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-0/wal
uuid: "8068030ce1d8484eb339c8264310627c"
format_stamp: "Formatted at 2026-08-08 18:32:43 on dist-test-slave-gp6n"
server_key: "b204a7f54f8e03ea4b111e96d0d16c05"
server_key_iv: "0bbb0f465f8c302ea7e6c4edc077a006"
server_key_version: "encryptionkey@0"
I20260808 18:32:43.893850 19444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:43.919404 19444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:43.920166 19444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:43.920377 19444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:43.920991 19444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:32:43.921943 19444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:32:43.922014 19444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:43.922089 19444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:32:43.922133 19444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:43.931769 19444 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:46659
I20260808 18:32:43.931797 19593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:46659 every 8 connection(s)
I20260808 18:32:43.932722 19444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-0/data/info.pb
I20260808 18:32:43.940224 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 19444
I20260808 18:32:43.940382 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-0/wal/instance
I20260808 18:32:43.940701 30010 external_mini_cluster.cc:1428] Setting key 982e8ddf65a429c0613b34bcfafb462f
I20260808 18:32:43.942740 19420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:43.934781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:59575 (local address 127.29.78.190:36579)
0808 18:32:43.934944 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:43.934948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:43.936065 (+ 1117us) server_negotiation.cc:407] Connection header received
0808 18:32:43.937539 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:43.937543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:43.937604 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:43.937709 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:43.938657 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.939204 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.940402 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.940695 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.940811 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:43.942140 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:43.942147 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:43.942152 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:43.942168 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:43.942192 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:43.942202 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:43.942281 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:43.942536 (+ 255us) server_negotiation.cc:299] Negotiation successful
0808 18:32:43.942602 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 18:32:43.943104 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36579
--builtin_ntp_servers=127.29.78.148:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 18:32:43.943693 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:43.935202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36579 (local address 127.29.78.129:59575)
0808 18:32:43.935899 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:43.935939 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:43.937301 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:43.937886 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:43.937893 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:43.937976 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:32:43.938503 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:43.938515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.939348 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.939352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:43.940260 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:43.940269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:43.940857 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:43.940861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:43.941038 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:43.941782 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:32:43.941785 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:32:43.941838 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:32:43.941914 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:32:43.942301 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:32:43.942311 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:32:43.942317 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 18:32:43.942533 (+ 216us) client_negotiation.cc:240] Negotiation successful
0808 18:32:43.942821 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":135,"threads_started":1}
I20260808 18:32:43.944901 19594 heartbeater.cc:347] Connected to a master server at 127.29.78.190:36579
I20260808 18:32:43.945176 19594 heartbeater.cc:464] Registering TS with master...
I20260808 18:32:43.945737 19594 heartbeater.cc:511] Master 127.29.78.190:36579 requested a full tablet report, sending...
I20260808 18:32:43.947258 19377 ts_manager.cc:194] Registered new tserver with Master: 8068030ce1d8484eb339c8264310627c (127.29.78.129:46659)
I20260808 18:32:43.948416 19377 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.129:59575
W20260808 18:32:44.056121 19597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:44.056411 19597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:44.056483 19597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:44.060505 19597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:44.060578 19597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:44.060660 19597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260808 18:32:44.065483 19597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:44.066550 19597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:44.067900 19597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:44.077551 19605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:44.077562 19602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:44.077553 19603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:44.077890 19597 server_base.cc:1034] running on GCE node
I20260808 18:32:44.078286 19597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:44.078960 19597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:44.080142 19597 hybrid_clock.cc:648] HybridClock initialized: now 1786213964080109 us; error 44 us; skew 500 ppm
I20260808 18:32:44.081832 19597 webserver.cc:460] Webserver started at http://127.29.78.130:36695/ using document root <none> and password file <none>
I20260808 18:32:44.082389 19597 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:44.082540 19597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:44.082770 19597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:44.084790 19597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-1/data/instance:
uuid: "abb473cb52ee47b4b8a11fe252e08cee"
format_stamp: "Formatted at 2026-08-08 18:32:44 on dist-test-slave-gp6n"
server_key: "2b9e46fddb04145b93092fba35164d64"
server_key_iv: "6ab2b2317e10b17cff8a94df7fa696d8"
server_key_version: "encryptionkey@0"
I20260808 18:32:44.085364 19597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "abb473cb52ee47b4b8a11fe252e08cee"
format_stamp: "Formatted at 2026-08-08 18:32:44 on dist-test-slave-gp6n"
server_key: "2b9e46fddb04145b93092fba35164d64"
server_key_iv: "6ab2b2317e10b17cff8a94df7fa696d8"
server_key_version: "encryptionkey@0"
I20260808 18:32:44.088963 19597 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260808 18:32:44.091399 19611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:44.092604 19597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:32:44.092705 19597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-1/wal
uuid: "abb473cb52ee47b4b8a11fe252e08cee"
format_stamp: "Formatted at 2026-08-08 18:32:44 on dist-test-slave-gp6n"
server_key: "2b9e46fddb04145b93092fba35164d64"
server_key_iv: "6ab2b2317e10b17cff8a94df7fa696d8"
server_key_version: "encryptionkey@0"
I20260808 18:32:44.092787 19597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:44.103082 19597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:44.103622 19597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:44.103761 19597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:44.104242 19597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:32:44.105068 19597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:32:44.105114 19597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:44.105152 19597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:32:44.105167 19597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:44.113996 19597 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:41611
I20260808 18:32:44.114019 19735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:41611 every 8 connection(s)
I20260808 18:32:44.114867 19597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-1/data/info.pb
I20260808 18:32:44.120540 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 19597
I20260808 18:32:44.120692 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-1/wal/instance
I20260808 18:32:44.121002 30010 external_mini_cluster.cc:1428] Setting key 01b46cd7f12e3e71b92305901f3c674e
I20260808 18:32:44.122999 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36579
--builtin_ntp_servers=127.29.78.148:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 18:32:44.124298 19420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:44.116601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:53593 (local address 127.29.78.190:36579)
0808 18:32:44.116735 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:44.116739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:44.117708 (+ 969us) server_negotiation.cc:407] Connection header received
0808 18:32:44.119116 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:44.119119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:44.119169 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:44.119252 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:44.120119 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.120692 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.122135 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.122502 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.122620 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:44.123780 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:44.123784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:44.123786 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:44.123800 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:44.123821 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:44.123838 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:44.123934 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:44.124098 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 18:32:44.124161 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 18:32:44.125118 19739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:44.116981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36579 (local address 127.29.78.130:53593)
0808 18:32:44.117586 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:44.117626 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:44.118900 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:44.119412 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:44.119419 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:44.119506 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:32:44.119970 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:44.119994 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.120864 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.120867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:44.122003 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:44.122012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.122661 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.122665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:44.122839 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:44.123518 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:32:44.123520 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:32:44.123560 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:32:44.123621 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:32:44.123938 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:32:44.123950 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:32:44.123959 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 18:32:44.124087 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 18:32:44.124362 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":111,"threads_started":1}
I20260808 18:32:44.126389 19736 heartbeater.cc:347] Connected to a master server at 127.29.78.190:36579
I20260808 18:32:44.126679 19736 heartbeater.cc:464] Registering TS with master...
I20260808 18:32:44.127192 19736 heartbeater.cc:511] Master 127.29.78.190:36579 requested a full tablet report, sending...
I20260808 18:32:44.128316 19377 ts_manager.cc:194] Registered new tserver with Master: abb473cb52ee47b4b8a11fe252e08cee (127.29.78.130:41611)
I20260808 18:32:44.129215 19377 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.130:53593
W20260808 18:32:44.239557 19740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:44.239797 19740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:44.239833 19740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:44.243726 19740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:44.243773 19740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:44.243826 19740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260808 18:32:44.248345 19740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:44.249286 19740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:44.251076 19740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:44.261020 19749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:44.261092 19745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:44.261020 19746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:44.261302 19740 server_base.cc:1034] running on GCE node
I20260808 18:32:44.261838 19740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:44.262554 19740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:44.263747 19740 hybrid_clock.cc:648] HybridClock initialized: now 1786213964263733 us; error 27 us; skew 500 ppm
I20260808 18:32:44.265678 19740 webserver.cc:460] Webserver started at http://127.29.78.131:44345/ using document root <none> and password file <none>
I20260808 18:32:44.266271 19740 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:44.266335 19740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:44.266628 19740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:44.268765 19740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-2/data/instance:
uuid: "56922423831941b1a53c34ac4160d138"
format_stamp: "Formatted at 2026-08-08 18:32:44 on dist-test-slave-gp6n"
server_key: "595859f75ad67e0a71f97b43f2324278"
server_key_iv: "6a0b5a9f1b077bdee1abe116115006bf"
server_key_version: "encryptionkey@0"
I20260808 18:32:44.269381 19740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "56922423831941b1a53c34ac4160d138"
format_stamp: "Formatted at 2026-08-08 18:32:44 on dist-test-slave-gp6n"
server_key: "595859f75ad67e0a71f97b43f2324278"
server_key_iv: "6a0b5a9f1b077bdee1abe116115006bf"
server_key_version: "encryptionkey@0"
I20260808 18:32:44.273166 19740 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260808 18:32:44.275758 19755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:44.277021 19740 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:32:44.277150 19740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-2/wal
uuid: "56922423831941b1a53c34ac4160d138"
format_stamp: "Formatted at 2026-08-08 18:32:44 on dist-test-slave-gp6n"
server_key: "595859f75ad67e0a71f97b43f2324278"
server_key_iv: "6a0b5a9f1b077bdee1abe116115006bf"
server_key_version: "encryptionkey@0"
I20260808 18:32:44.277263 19740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:44.291056 19740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:44.291765 19740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:44.291980 19740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:44.292538 19740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:32:44.293445 19740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:32:44.293495 19740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:44.293537 19740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:32:44.293555 19740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:44.303197 19740 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:37207
I20260808 18:32:44.303226 19881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:37207 every 8 connection(s)
I20260808 18:32:44.304188 19740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-2/data/info.pb
I20260808 18:32:44.310906 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 19740
I20260808 18:32:44.311039 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-2/wal/instance
I20260808 18:32:44.311331 30010 external_mini_cluster.cc:1428] Setting key 737273dd70fc54205bd35169d8186852
I20260808 18:32:44.314270 19420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:44.306188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:47517 (local address 127.29.78.190:36579)
0808 18:32:44.306331 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:44.306335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:44.307538 (+ 1203us) server_negotiation.cc:407] Connection header received
0808 18:32:44.309190 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:44.309193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:44.309254 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:44.309372 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:44.310297 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.310854 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.312277 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.312596 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.312711 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:44.313863 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:44.313866 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:44.313868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:44.313878 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:44.313897 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:44.313916 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:44.313975 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:44.314118 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 18:32:44.314164 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 18:32:44.315115 19884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:44.306669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36579 (local address 127.29.78.131:47517)
0808 18:32:44.307383 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:44.307434 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:44.308972 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:44.309540 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:44.309547 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:44.309660 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:32:44.310158 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:44.310170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.310992 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.310995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:44.312154 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:44.312167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.312766 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.312771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:44.312971 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:44.313622 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:32:44.313637 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:32:44.313677 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:32:44.313735 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:32:44.313992 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:32:44.313999 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:32:44.314002 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 18:32:44.314102 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 18:32:44.314306 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":144,"threads_started":1}
I20260808 18:32:44.316161 19882 heartbeater.cc:347] Connected to a master server at 127.29.78.190:36579
I20260808 18:32:44.316392 19882 heartbeater.cc:464] Registering TS with master...
I20260808 18:32:44.316861 19882 heartbeater.cc:511] Master 127.29.78.190:36579 requested a full tablet report, sending...
I20260808 18:32:44.317792 19377 ts_manager.cc:194] Registered new tserver with Master: 56922423831941b1a53c34ac4160d138 (127.29.78.131:37207)
I20260808 18:32:44.318328 19377 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.131:47517
I20260808 18:32:44.325306 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:32:44.331256 19420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:44.327345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35252 (local address 127.29.78.190:36579)
0808 18:32:44.327538 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:44.327543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:44.327692 (+ 149us) server_negotiation.cc:407] Connection header received
0808 18:32:44.327854 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:44.327856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:44.327898 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:44.327985 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:44.328261 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.328765 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.329636 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.329879 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.329988 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:44.330802 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:44.330806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:44.330808 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:44.330820 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:44.330854 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:44.330862 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:44.330941 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:44.331071 (+ 130us) server_negotiation.cc:299] Negotiation successful
0808 18:32:44.331124 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 18:32:44.334134 19377 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:32:44.337220 19377 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:32:44.350975 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:44.345390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:37207 (local address 127.0.0.1:35426)
0808 18:32:44.345763 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:44.345808 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:44.345926 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:44.346803 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:44.346810 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:44.346834 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:44.346981 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:44.346988 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.348352 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.348356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:44.349366 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:44.349375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.350453 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.350458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:44.350553 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:44.350611 (+ 58us) client_negotiation.cc:769] Sending connection context
0808 18:32:44.350698 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 18:32:44.350816 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":83,"threads_started":1}
I20260808 18:32:44.351727 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:44.345414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:41611 (local address 127.0.0.1:38950)
0808 18:32:44.346234 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:44.346249 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:44.346336 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:44.347358 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:44.347362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:44.347392 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:44.347515 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:44.347525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.349114 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.349118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:44.350262 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:44.350273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.351306 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.351310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:44.351414 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:44.351435 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 18:32:44.351480 (+ 45us) client_negotiation.cc:240] Negotiation successful
0808 18:32:44.351576 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":1,"spinlock_wait_cycles":8064,"thread_start_us":119,"threads_started":1}
I20260808 18:32:44.352041 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:44.345569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35426 (local address 127.29.78.131:37207)
0808 18:32:44.346237 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:44.346252 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:44.346282 (+ 30us) server_negotiation.cc:407] Connection header received
0808 18:32:44.346363 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:44.346371 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:44.346627 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:44.347129 (+ 502us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:44.347267 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.348236 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.349507 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.350296 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.351353 (+ 1057us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:44.351555 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:44.351605 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 18:32:44.351705 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":144,"threads_started":1}
I20260808 18:32:44.352365 19899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:44.346108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38950 (local address 127.29.78.130:41611)
0808 18:32:44.346896 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:44.346907 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:44.346929 (+ 22us) server_negotiation.cc:407] Connection header received
0808 18:32:44.347014 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:44.347022 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:44.347204 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:44.347360 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:44.347665 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.348979 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.350433 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.351159 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.351872 (+ 713us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:44.352064 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:44.352118 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 18:32:44.352215 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":115,"threads_started":1}
I20260808 18:32:44.352629 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:44.346133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:46659 (local address 127.0.0.1:47844)
0808 18:32:44.346551 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:44.346564 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:44.346637 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:44.347599 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:44.347603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:44.347620 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:44.347709 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:44.347715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.349353 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.349357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:44.350513 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:44.350524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.351591 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.351595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:44.351688 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:44.351711 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 18:32:44.352235 (+ 524us) client_negotiation.cc:240] Negotiation successful
0808 18:32:44.352312 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260808 18:32:44.352442 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:44.346614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47844 (local address 127.29.78.129:46659)
0808 18:32:44.347119 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:44.347132 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:44.347158 (+ 26us) server_negotiation.cc:407] Connection header received
0808 18:32:44.347236 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:44.347240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:44.347434 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:44.347608 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:44.347807 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.349026 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.350650 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.351447 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.351563 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:44.351729 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:44.352150 (+ 421us) server_negotiation.cc:299] Negotiation successful
0808 18:32:44.352269 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":157,"threads_started":1}
I20260808 18:32:44.355103 19810 tablet_service.cc:1467] Processing CreateTablet for tablet c6bb88d211134c7aac687321c4838771 (DEFAULT_TABLE table=test-table [id=b96964be38b348609badb7e52a552b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:32:44.355103 19664 tablet_service.cc:1467] Processing CreateTablet for tablet c6bb88d211134c7aac687321c4838771 (DEFAULT_TABLE table=test-table [id=b96964be38b348609badb7e52a552b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:32:44.355474 19518 tablet_service.cc:1467] Processing CreateTablet for tablet c6bb88d211134c7aac687321c4838771 (DEFAULT_TABLE table=test-table [id=b96964be38b348609badb7e52a552b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:32:44.356242 19664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6bb88d211134c7aac687321c4838771. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:44.356242 19810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6bb88d211134c7aac687321c4838771. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:44.356542 19518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6bb88d211134c7aac687321c4838771. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:44.362707 19901 tablet_bootstrap.cc:492] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c: Bootstrap starting.
I20260808 18:32:44.364463 19903 tablet_bootstrap.cc:492] T c6bb88d211134c7aac687321c4838771 P 56922423831941b1a53c34ac4160d138: Bootstrap starting.
I20260808 18:32:44.364706 19902 tablet_bootstrap.cc:492] T c6bb88d211134c7aac687321c4838771 P abb473cb52ee47b4b8a11fe252e08cee: Bootstrap starting.
I20260808 18:32:44.365236 19901 tablet_bootstrap.cc:654] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:44.366110 19901 log.cc:824] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:44.366648 19902 tablet_bootstrap.cc:654] T c6bb88d211134c7aac687321c4838771 P abb473cb52ee47b4b8a11fe252e08cee: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:44.366648 19903 tablet_bootstrap.cc:654] T c6bb88d211134c7aac687321c4838771 P 56922423831941b1a53c34ac4160d138: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:44.367491 19902 log.cc:824] T c6bb88d211134c7aac687321c4838771 P abb473cb52ee47b4b8a11fe252e08cee: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:44.367487 19903 log.cc:824] T c6bb88d211134c7aac687321c4838771 P 56922423831941b1a53c34ac4160d138: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:44.368383 19901 tablet_bootstrap.cc:492] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c: No bootstrap required, opened a new log
I20260808 18:32:44.368631 19901 ts_tablet_manager.cc:1397] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:32:44.369256 19902 tablet_bootstrap.cc:492] T c6bb88d211134c7aac687321c4838771 P abb473cb52ee47b4b8a11fe252e08cee: No bootstrap required, opened a new log
I20260808 18:32:44.369479 19902 ts_tablet_manager.cc:1397] T c6bb88d211134c7aac687321c4838771 P abb473cb52ee47b4b8a11fe252e08cee: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 18:32:44.369899 19903 tablet_bootstrap.cc:492] T c6bb88d211134c7aac687321c4838771 P 56922423831941b1a53c34ac4160d138: No bootstrap required, opened a new log
I20260808 18:32:44.370093 19903 ts_tablet_manager.cc:1397] T c6bb88d211134c7aac687321c4838771 P 56922423831941b1a53c34ac4160d138: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:32:44.372607 19902 raft_consensus.cc:348] T c6bb88d211134c7aac687321c4838771 P abb473cb52ee47b4b8a11fe252e08cee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56922423831941b1a53c34ac4160d138" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37207 } } peers { permanent_uuid: "abb473cb52ee47b4b8a11fe252e08cee" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41611 } } peers { permanent_uuid: "8068030ce1d8484eb339c8264310627c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46659 } }
I20260808 18:32:44.372845 19902 raft_consensus.cc:374] T c6bb88d211134c7aac687321c4838771 P abb473cb52ee47b4b8a11fe252e08cee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:44.372907 19902 raft_consensus.cc:729] T c6bb88d211134c7aac687321c4838771 P abb473cb52ee47b4b8a11fe252e08cee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb473cb52ee47b4b8a11fe252e08cee, State: Initialized, Role: FOLLOWER
I20260808 18:32:44.373440 19902 consensus_queue.cc:260] T c6bb88d211134c7aac687321c4838771 P abb473cb52ee47b4b8a11fe252e08cee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56922423831941b1a53c34ac4160d138" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37207 } } peers { permanent_uuid: "abb473cb52ee47b4b8a11fe252e08cee" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41611 } } peers { permanent_uuid: "8068030ce1d8484eb339c8264310627c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46659 } }
I20260808 18:32:44.373449 19901 raft_consensus.cc:348] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56922423831941b1a53c34ac4160d138" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37207 } } peers { permanent_uuid: "abb473cb52ee47b4b8a11fe252e08cee" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41611 } } peers { permanent_uuid: "8068030ce1d8484eb339c8264310627c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46659 } }
I20260808 18:32:44.373710 19901 raft_consensus.cc:374] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:44.373767 19901 raft_consensus.cc:729] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8068030ce1d8484eb339c8264310627c, State: Initialized, Role: FOLLOWER
I20260808 18:32:44.374222 19901 consensus_queue.cc:260] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56922423831941b1a53c34ac4160d138" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37207 } } peers { permanent_uuid: "abb473cb52ee47b4b8a11fe252e08cee" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41611 } } peers { permanent_uuid: "8068030ce1d8484eb339c8264310627c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46659 } }
I20260808 18:32:44.374468 19902 ts_tablet_manager.cc:1428] T c6bb88d211134c7aac687321c4838771 P abb473cb52ee47b4b8a11fe252e08cee: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 18:32:44.374835 19736 heartbeater.cc:503] Master 127.29.78.190:36579 was elected leader, sending a full tablet report...
I20260808 18:32:44.374890 19903 raft_consensus.cc:348] T c6bb88d211134c7aac687321c4838771 P 56922423831941b1a53c34ac4160d138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56922423831941b1a53c34ac4160d138" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37207 } } peers { permanent_uuid: "abb473cb52ee47b4b8a11fe252e08cee" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41611 } } peers { permanent_uuid: "8068030ce1d8484eb339c8264310627c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46659 } }
I20260808 18:32:44.375133 19903 raft_consensus.cc:374] T c6bb88d211134c7aac687321c4838771 P 56922423831941b1a53c34ac4160d138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:44.375186 19903 raft_consensus.cc:729] T c6bb88d211134c7aac687321c4838771 P 56922423831941b1a53c34ac4160d138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56922423831941b1a53c34ac4160d138, State: Initialized, Role: FOLLOWER
I20260808 18:32:44.375245 19594 heartbeater.cc:503] Master 127.29.78.190:36579 was elected leader, sending a full tablet report...
I20260808 18:32:44.375274 19901 ts_tablet_manager.cc:1428] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260808 18:32:44.375650 19903 consensus_queue.cc:260] T c6bb88d211134c7aac687321c4838771 P 56922423831941b1a53c34ac4160d138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56922423831941b1a53c34ac4160d138" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37207 } } peers { permanent_uuid: "abb473cb52ee47b4b8a11fe252e08cee" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41611 } } peers { permanent_uuid: "8068030ce1d8484eb339c8264310627c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46659 } }
I20260808 18:32:44.376298 19882 heartbeater.cc:503] Master 127.29.78.190:36579 was elected leader, sending a full tablet report...
I20260808 18:32:44.376612 19903 ts_tablet_manager.cc:1428] T c6bb88d211134c7aac687321c4838771 P 56922423831941b1a53c34ac4160d138: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260808 18:32:44.432659 19908 raft_consensus.cc:482] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:32:44.432875 19908 raft_consensus.cc:504] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56922423831941b1a53c34ac4160d138" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37207 } } peers { permanent_uuid: "abb473cb52ee47b4b8a11fe252e08cee" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41611 } } peers { permanent_uuid: "8068030ce1d8484eb339c8264310627c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46659 } }
W20260808 18:32:44.433878 19595 tablet.cc:2369] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:32:44.434206 19908 leader_election.cc:288] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56922423831941b1a53c34ac4160d138 (127.29.78.131:37207), abb473cb52ee47b4b8a11fe252e08cee (127.29.78.130:41611)
I20260808 18:32:44.439193 19899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:44.434771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:46449 (local address 127.29.78.130:41611)
0808 18:32:44.434906 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:44.434910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:44.434949 (+ 39us) server_negotiation.cc:407] Connection header received
0808 18:32:44.435054 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:44.435060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:44.435105 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:44.435180 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:44.435456 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.436199 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.437894 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.438506 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.438625 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:44.438674 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:44.438998 (+ 324us) server_negotiation.cc:299] Negotiation successful
0808 18:32:44.439065 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 18:32:44.439193 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:44.434593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:41347 (local address 127.29.78.131:37207)
0808 18:32:44.434719 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:44.434724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:44.434742 (+ 18us) server_negotiation.cc:407] Connection header received
0808 18:32:44.434916 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:44.434922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:44.434983 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:44.435067 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:32:44.435373 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.436198 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.437894 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.438505 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.438627 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:44.438674 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:44.438993 (+ 319us) server_negotiation.cc:299] Negotiation successful
0808 18:32:44.439067 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 18:32:44.439203 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:44.434457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:37207 (local address 127.29.78.129:41347)
0808 18:32:44.434663 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:44.434680 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:44.434771 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:44.435104 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:44.435106 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:44.435126 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:44.435250 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:44.435257 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.436328 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.436332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:44.437761 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:44.437774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.438644 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.438646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:44.438828 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:44.438878 (+ 50us) client_negotiation.cc:769] Sending connection context
0808 18:32:44.438964 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 18:32:44.439057 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":173}
I20260808 18:32:44.439210 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:44.434497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:41611 (local address 127.29.78.129:46449)
0808 18:32:44.434878 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:44.434892 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:32:44.434974 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:32:44.435203 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:32:44.435206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:32:44.435218 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:32:44.435334 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:44.435339 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.436328 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.436332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:44.437767 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:32:44.437774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.438639 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.438643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:32:44.438818 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:44.438874 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 18:32:44.438960 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 18:32:44.439057 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":138,"thread_start_us":105,"threads_started":1}
I20260808 18:32:44.439924 19832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6bb88d211134c7aac687321c4838771" candidate_uuid: "8068030ce1d8484eb339c8264310627c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56922423831941b1a53c34ac4160d138" is_pre_election: true
I20260808 18:32:44.440166 19684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6bb88d211134c7aac687321c4838771" candidate_uuid: "8068030ce1d8484eb339c8264310627c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb473cb52ee47b4b8a11fe252e08cee" is_pre_election: true
I20260808 18:32:44.440229 19832 raft_consensus.cc:2454] T c6bb88d211134c7aac687321c4838771 P 56922423831941b1a53c34ac4160d138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8068030ce1d8484eb339c8264310627c in term 0.
I20260808 18:32:44.440430 19684 raft_consensus.cc:2454] T c6bb88d211134c7aac687321c4838771 P abb473cb52ee47b4b8a11fe252e08cee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8068030ce1d8484eb339c8264310627c in term 0.
I20260808 18:32:44.440752 19464 leader_election.cc:302] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56922423831941b1a53c34ac4160d138, 8068030ce1d8484eb339c8264310627c; no voters:
I20260808 18:32:44.441026 19908 raft_consensus.cc:2781] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:32:44.441136 19908 raft_consensus.cc:482] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:32:44.441191 19908 raft_consensus.cc:3037] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:44.442302 19908 raft_consensus.cc:504] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56922423831941b1a53c34ac4160d138" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37207 } } peers { permanent_uuid: "abb473cb52ee47b4b8a11fe252e08cee" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41611 } } peers { permanent_uuid: "8068030ce1d8484eb339c8264310627c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46659 } }
I20260808 18:32:44.442688 19908 leader_election.cc:288] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c [CANDIDATE]: Term 1 election: Requested vote from peers 56922423831941b1a53c34ac4160d138 (127.29.78.131:37207), abb473cb52ee47b4b8a11fe252e08cee (127.29.78.130:41611)
I20260808 18:32:44.443010 19684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6bb88d211134c7aac687321c4838771" candidate_uuid: "8068030ce1d8484eb339c8264310627c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb473cb52ee47b4b8a11fe252e08cee"
I20260808 18:32:44.442984 19832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6bb88d211134c7aac687321c4838771" candidate_uuid: "8068030ce1d8484eb339c8264310627c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56922423831941b1a53c34ac4160d138"
I20260808 18:32:44.443125 19684 raft_consensus.cc:3037] T c6bb88d211134c7aac687321c4838771 P abb473cb52ee47b4b8a11fe252e08cee [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:44.443125 19832 raft_consensus.cc:3037] T c6bb88d211134c7aac687321c4838771 P 56922423831941b1a53c34ac4160d138 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:44.444388 19684 raft_consensus.cc:2454] T c6bb88d211134c7aac687321c4838771 P abb473cb52ee47b4b8a11fe252e08cee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8068030ce1d8484eb339c8264310627c in term 1.
I20260808 18:32:44.444388 19832 raft_consensus.cc:2454] T c6bb88d211134c7aac687321c4838771 P 56922423831941b1a53c34ac4160d138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8068030ce1d8484eb339c8264310627c in term 1.
I20260808 18:32:44.444690 19462 leader_election.cc:302] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8068030ce1d8484eb339c8264310627c, abb473cb52ee47b4b8a11fe252e08cee; no voters:
I20260808 18:32:44.444847 19908 raft_consensus.cc:2781] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:32:44.445091 19908 raft_consensus.cc:686] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c [term 1 LEADER]: Becoming Leader. State: Replica: 8068030ce1d8484eb339c8264310627c, State: Running, Role: LEADER
I20260808 18:32:44.445423 19908 consensus_queue.cc:237] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56922423831941b1a53c34ac4160d138" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37207 } } peers { permanent_uuid: "abb473cb52ee47b4b8a11fe252e08cee" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41611 } } peers { permanent_uuid: "8068030ce1d8484eb339c8264310627c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46659 } }
I20260808 18:32:44.448567 19376 catalog_manager.cc:5495] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8068030ce1d8484eb339c8264310627c (127.29.78.129). New cstate: current_term: 1 leader_uuid: "8068030ce1d8484eb339c8264310627c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56922423831941b1a53c34ac4160d138" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abb473cb52ee47b4b8a11fe252e08cee" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8068030ce1d8484eb339c8264310627c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46659 } health_report { overall_health: HEALTHY } } }
I20260808 18:32:44.478879 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:44.475160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.29.78.129:46659)
0808 18:32:44.475291 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:44.475295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:44.475308 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:32:44.475451 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:44.475454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:44.475513 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:44.475606 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:32:44.475832 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.476341 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.477262 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:44.477526 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:44.477610 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:44.477984 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:32:44.478390 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:32:44.478564 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:44.478663 (+ 99us) server_negotiation.cc:299] Negotiation successful
0808 18:32:44.478715 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 18:32:44.484673 19684 raft_consensus.cc:1261] T c6bb88d211134c7aac687321c4838771 P abb473cb52ee47b4b8a11fe252e08cee [term 1 FOLLOWER]: Refusing update from remote peer 8068030ce1d8484eb339c8264310627c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:32:44.484670 19832 raft_consensus.cc:1261] T c6bb88d211134c7aac687321c4838771 P 56922423831941b1a53c34ac4160d138 [term 1 FOLLOWER]: Refusing update from remote peer 8068030ce1d8484eb339c8264310627c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:32:44.485353 19913 consensus_queue.cc:1035] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c [LEADER]: Connected to new peer: Peer: permanent_uuid: "56922423831941b1a53c34ac4160d138" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:32:44.485508 19908 consensus_queue.cc:1035] T c6bb88d211134c7aac687321c4838771 P 8068030ce1d8484eb339c8264310627c [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb473cb52ee47b4b8a11fe252e08cee" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:32:44.492683 19916 mvcc.cc:205] Tried to move back new op lower bound from 7316332398522388480 to 7316332398371299328. Current Snapshot: MvccSnapshot[applied={T|T < 7316332398522388480}]
I20260808 18:32:44.496623 19918 mvcc.cc:205] Tried to move back new op lower bound from 7316332398522388480 to 7316332398371299328. Current Snapshot: MvccSnapshot[applied={T|T < 7316332398522388480 or (T in {7316332398522388480})}]
I20260808 18:32:44.497054 19917 mvcc.cc:205] Tried to move back new op lower bound from 7316332398522388480 to 7316332398371299328. Current Snapshot: MvccSnapshot[applied={T|T < 7316332398522388480 or (T in {7316332398522388480})}]
I20260808 18:32:44.502038 19377 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:32:44.502175 19377 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:32:44.504648 19377 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e842fcc228149708230f1372bca4d18: Sending DeleteTablet for 3 replicas of tablet c6bb88d211134c7aac687321c4838771
I20260808 18:32:44.505395 19664 tablet_service.cc:1514] Processing DeleteTablet for tablet c6bb88d211134c7aac687321c4838771 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:32:44 UTC) from {username='slave'} at 127.0.0.1:38950
I20260808 18:32:44.505604 19518 tablet_service.cc:1514] Processing DeleteTablet for tablet c6bb88d211134c7aac687321c4838771 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:32:44 UTC) from {username='slave'} at 127.0.0.1:47844
I20260808 18:32:44.505615 19810 tablet_service.cc:1514] Processing DeleteTablet for tablet c6bb88d211134c7aac687321c4838771 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:32:44 UTC) from {username='slave'} at 127.0.0.1:35426
I20260808 18:32:44.505920 19926 tablet_replica.cc:330] T c6bb88d211134c7aac687321c4838771 P abb473cb52ee47b4b8a11fe252e08cee: stopping tablet replica
I20260808 18:32:44.506224 19926 raft_consensus.cc:2229] T c6bb88d211134c7aac687321c4838771 P abb473cb52ee47b4b8a11fe252e08cee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:32:44.506500 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 19444
I20260808 18:32:44.506601 19926 raft_consensus.cc:2258] T c6bb88d211134c7aac687321c4838771 P abb473cb52ee47b4b8a11fe252e08cee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:32:44.509162 19926 ts_tablet_manager.cc:1905] T c6bb88d211134c7aac687321c4838771 P abb473cb52ee47b4b8a11fe252e08cee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:32:44.510025 19927 tablet_replica.cc:330] T c6bb88d211134c7aac687321c4838771 P 56922423831941b1a53c34ac4160d138: stopping tablet replica
I20260808 18:32:44.510314 19927 raft_consensus.cc:2229] T c6bb88d211134c7aac687321c4838771 P 56922423831941b1a53c34ac4160d138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:32:44.510625 19927 raft_consensus.cc:2258] T c6bb88d211134c7aac687321c4838771 P 56922423831941b1a53c34ac4160d138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:32:44.511868 19927 ts_tablet_manager.cc:1905] T c6bb88d211134c7aac687321c4838771 P 56922423831941b1a53c34ac4160d138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:32:44.512748 19926 ts_tablet_manager.cc:1918] T c6bb88d211134c7aac687321c4838771 P abb473cb52ee47b4b8a11fe252e08cee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:32:44.512850 19926 log.cc:1196] T c6bb88d211134c7aac687321c4838771 P abb473cb52ee47b4b8a11fe252e08cee: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-1/wal/wals/c6bb88d211134c7aac687321c4838771
I20260808 18:32:44.513175 19926 ts_tablet_manager.cc:1939] T c6bb88d211134c7aac687321c4838771 P abb473cb52ee47b4b8a11fe252e08cee: Deleting consensus metadata
I20260808 18:32:44.514125 19927 ts_tablet_manager.cc:1918] T c6bb88d211134c7aac687321c4838771 P 56922423831941b1a53c34ac4160d138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:32:44.514235 19927 log.cc:1196] T c6bb88d211134c7aac687321c4838771 P 56922423831941b1a53c34ac4160d138: Deleting WAL directory at /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786213708769829-30010-0/minicluster-data/ts-2/wal/wals/c6bb88d211134c7aac687321c4838771
I20260808 18:32:44.514623 19927 ts_tablet_manager.cc:1939] T c6bb88d211134c7aac687321c4838771 P 56922423831941b1a53c34ac4160d138: Deleting consensus metadata
I20260808 18:32:44.514598 19359 catalog_manager.cc:4841] TS abb473cb52ee47b4b8a11fe252e08cee (127.29.78.130:41611): tablet c6bb88d211134c7aac687321c4838771 (table test-table [id=b96964be38b348609badb7e52a552b70]) successfully deleted
I20260808 18:32:44.515676 19361 catalog_manager.cc:4841] TS 56922423831941b1a53c34ac4160d138 (127.29.78.131:37207): tablet c6bb88d211134c7aac687321c4838771 (table test-table [id=b96964be38b348609badb7e52a552b70]) successfully deleted
W20260808 18:32:44.516731 19361 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:46659 (error 108)
I20260808 18:32:44.516860 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 19597
I20260808 18:32:44.519624 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:44.519246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:46659 (local address 127.0.0.1:47856)
0808 18:32:44.519477 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 18:32:44.519554 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:46659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260808 18:32:44.519816 19361 catalog_manager.cc:4584] TS 8068030ce1d8484eb339c8264310627c (127.29.78.129:46659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6bb88d211134c7aac687321c4838771: Network error: Client connection negotiation failed: client connection to 127.29.78.129:46659: connect: Connection refused (error 111)
I20260808 18:32:44.524765 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 19740
I20260808 18:32:44.532258 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 19341
2026-08-08T18:32:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1047 ms)
[----------] 6 tests from EncryptionPolicyTest (10775 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:32:44 dist-test-slave-gp6n krb5kdc[19933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:32:44 dist-test-slave-gp6n krb5kdc[19933](info): set up 2 sockets
Aug 08 18:32:44 dist-test-slave-gp6n krb5kdc[19933](info): commencing operation
krb5kdc: starting...
W20260808 18:32:45.625523 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:32:45 dist-test-slave-gp6n krb5kdc[19933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:32:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:32:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:32:45.757815 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44949
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:45.871173 19947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:45.871450 19947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:45.871493 19947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:45.875566 19947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:32:45.875641 19947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:45.875661 19947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:45.875679 19947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:32:45.875694 19947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:32:45.881069 19947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44949
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:45.882139 19947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:45.883193 19947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:45.892082 19958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:45.892072 19954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:45.892073 19955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:45.892297 19947 server_base.cc:1034] running on GCE node
I20260808 18:32:45.893018 19947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:45.894114 19947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:45.895306 19947 hybrid_clock.cc:648] HybridClock initialized: now 1786213965895264 us; error 56 us; skew 500 ppm
Aug 08 18:32:45 dist-test-slave-gp6n krb5kdc[19933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:45.898042 19947 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:32:45.898980 19947 webserver.cc:460] Webserver started at http://127.0.0.1:38073/ using document root <none> and password file <none>
I20260808 18:32:45.899549 19947 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:45.899643 19947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:45.899883 19947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:45.901942 19947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "23ccc2980417483eaec7dd58835c286b"
format_stamp: "Formatted at 2026-08-08 18:32:45 on dist-test-slave-gp6n"
server_key: "65ac18fdf60523903d973a00b7d23f4b"
server_key_iv: "f89e22006af33e71e79e21d8011c78fd"
server_key_version: "encryptionkey@0"
I20260808 18:32:45.902590 19947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "23ccc2980417483eaec7dd58835c286b"
format_stamp: "Formatted at 2026-08-08 18:32:45 on dist-test-slave-gp6n"
server_key: "65ac18fdf60523903d973a00b7d23f4b"
server_key_iv: "f89e22006af33e71e79e21d8011c78fd"
server_key_version: "encryptionkey@0"
I20260808 18:32:45.906248 19947 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:32:45.908597 19965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:45.909750 19947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:32:45.909891 19947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "23ccc2980417483eaec7dd58835c286b"
format_stamp: "Formatted at 2026-08-08 18:32:45 on dist-test-slave-gp6n"
server_key: "65ac18fdf60523903d973a00b7d23f4b"
server_key_iv: "f89e22006af33e71e79e21d8011c78fd"
server_key_version: "encryptionkey@0"
I20260808 18:32:45.910012 19947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:45.930226 19947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:45.933547 19947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:45.933774 19947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:45.941285 19947 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44949
I20260808 18:32:45.941313 20022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44949 every 8 connection(s)
I20260808 18:32:45.942227 19947 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:32:45.944905 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 19947
I20260808 18:32:45.945041 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:32:45.945117 20023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:45.945369 30010 external_mini_cluster.cc:1428] Setting key 4f8632d7dc2f09ba17bd102a9df81561
I20260808 18:32:45.951542 20023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b: Bootstrap starting.
I20260808 18:32:45.953912 20023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b: Neither blocks nor log segments found. Creating new log.
Aug 08 18:32:45 dist-test-slave-gp6n krb5kdc[19933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 18:32:45.954766 20023 log.cc:824] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:45.957070 20023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b: No bootstrap required, opened a new log
I20260808 18:32:45.960641 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:45.946905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51741 (local address 127.0.0.1:44949)
0808 18:32:45.947356 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:45.947370 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:45.947404 (+ 34us) server_negotiation.cc:407] Connection header received
0808 18:32:45.948334 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:45.948352 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:45.948678 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:45.949020 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:45.949986 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:45.951054 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:45.951997 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:45.952466 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:45.952639 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:45.955364 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:45.955368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:45.955380 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:45.955427 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:45.957434 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:45.957925 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:45.957931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:45.957936 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:45.958073 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:45.958384 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:45.958387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:45.958389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:45.958922 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:45.959153 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:45.959447 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 18:32:45.959698 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":153,"threads_started":1}
I20260808 18:32:45.961627 20023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ccc2980417483eaec7dd58835c286b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44949 } }
I20260808 18:32:45.961915 20023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:45.961997 20023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ccc2980417483eaec7dd58835c286b, State: Initialized, Role: FOLLOWER
I20260808 18:32:45.962664 20023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ccc2980417483eaec7dd58835c286b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44949 } }
I20260808 18:32:45.962849 20023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:32:45.962936 20023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:32:45.963032 20023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:45.964164 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:32:45.964207 20023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ccc2980417483eaec7dd58835c286b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44949 } }
I20260808 18:32:45.964557 20023 leader_election.cc:302] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23ccc2980417483eaec7dd58835c286b; no voters:
I20260808 18:32:45.964874 20023 leader_election.cc:288] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:32:45.965034 20029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:32:45.965250 20029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b [term 1 LEADER]: Becoming Leader. State: Replica: 23ccc2980417483eaec7dd58835c286b, State: Running, Role: LEADER
I20260808 18:32:45.965570 20029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ccc2980417483eaec7dd58835c286b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44949 } }
I20260808 18:32:45.966126 20023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:32:45.966817 20029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23ccc2980417483eaec7dd58835c286b. Latest consensus state: current_term: 1 leader_uuid: "23ccc2980417483eaec7dd58835c286b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ccc2980417483eaec7dd58835c286b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44949 } } }
I20260808 18:32:45.966955 20029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:45.967358 20040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:32:45.967295 20031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23ccc2980417483eaec7dd58835c286b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ccc2980417483eaec7dd58835c286b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44949 } } }
I20260808 18:32:45.967418 20031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:45.970391 20040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:32:45.976598 20040 catalog_manager.cc:1347] Generated new cluster ID: 771799749218461b92dc735e150af74e
I20260808 18:32:45.976652 20040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:32:45.976795 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:45.966434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37275 (local address 127.0.0.1:44949)
0808 18:32:45.966554 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:45.966558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:45.967063 (+ 505us) server_negotiation.cc:407] Connection header received
0808 18:32:45.967263 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:45.967266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:45.967311 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:45.967362 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:45.968303 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:45.969067 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:45.970795 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:45.971134 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:45.971197 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:45.972841 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:45.972845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:45.972848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:45.972875 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:45.974631 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:45.975220 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:45.975224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:45.975225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:45.975323 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:45.975731 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:45.975734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:45.975736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:45.975983 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:45.976142 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:45.976523 (+ 381us) server_negotiation.cc:299] Negotiation successful
0808 18:32:45.976631 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 18:32:45.983857 20040 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:32:45.984822 20040 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:32:45.993360 20040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23ccc2980417483eaec7dd58835c286b: Generated new TSK 0
I20260808 18:32:45.994035 20040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:32:46.004145 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 19947
W20260808 18:32:46.016958 19930 connection.cc:535] client connection to 127.0.0.1:44949 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44949): Connection reset by peer (error 104)
2026-08-08T18:32:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1476 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:32:46 dist-test-slave-gp6n krb5kdc[20055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:32:46 dist-test-slave-gp6n krb5kdc[20055](info): set up 2 sockets
Aug 08 18:32:46 dist-test-slave-gp6n krb5kdc[20055](info): commencing operation
krb5kdc: starting...
W20260808 18:32:47.110735 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:32:47 dist-test-slave-gp6n krb5kdc[20055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:32:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:32:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:32:47.254714 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:47.368621 20070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:47.369009 20070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:47.369125 20070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:47.376277 20070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:32:47.376351 20070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:47.376372 20070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:47.376390 20070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:32:47.376405 20070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:32:47.381906 20070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36775
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:47.383082 20070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:47.384177 20070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:47.393143 20077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:47.393095 20079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:47.393098 20076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:47.393265 20070 server_base.cc:1034] running on GCE node
I20260808 18:32:47.393899 20070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:47.395087 20070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:47.396283 20070 hybrid_clock.cc:648] HybridClock initialized: now 1786213967396254 us; error 42 us; skew 500 ppm
Aug 08 18:32:47 dist-test-slave-gp6n krb5kdc[20055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:47.399235 20070 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:32:47.400126 20070 webserver.cc:460] Webserver started at http://127.0.0.1:38253/ using document root <none> and password file <none>
I20260808 18:32:47.400637 20070 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:47.400696 20070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:47.400859 20070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:47.403019 20070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "71db59fb1bab4a1bb3ca46057d605bad"
format_stamp: "Formatted at 2026-08-08 18:32:47 on dist-test-slave-gp6n"
server_key: "d4345925e26874b60356732ea40b8d80"
server_key_iv: "3cc66819285153566c176ca26308de20"
server_key_version: "encryptionkey@0"
I20260808 18:32:47.403558 20070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "71db59fb1bab4a1bb3ca46057d605bad"
format_stamp: "Formatted at 2026-08-08 18:32:47 on dist-test-slave-gp6n"
server_key: "d4345925e26874b60356732ea40b8d80"
server_key_iv: "3cc66819285153566c176ca26308de20"
server_key_version: "encryptionkey@0"
I20260808 18:32:47.407305 20070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 18:32:47.409619 20086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:47.410970 20070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:32:47.411072 20070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "71db59fb1bab4a1bb3ca46057d605bad"
format_stamp: "Formatted at 2026-08-08 18:32:47 on dist-test-slave-gp6n"
server_key: "d4345925e26874b60356732ea40b8d80"
server_key_iv: "3cc66819285153566c176ca26308de20"
server_key_version: "encryptionkey@0"
I20260808 18:32:47.411156 20070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:47.438144 20070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:47.442210 20070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:47.442494 20070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:47.450471 20070 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36775
I20260808 18:32:47.450546 20146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36775 every 8 connection(s)
I20260808 18:32:47.451577 20070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:32:47.452356 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 20070
I20260808 18:32:47.452510 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:32:47.452845 30010 external_mini_cluster.cc:1428] Setting key fe1e730fc8425e9c297c59048e21a7aa
I20260808 18:32:47.454941 20148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:47.460686 20148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad: Bootstrap starting.
I20260808 18:32:47.462815 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:47.454356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33585 (local address 127.0.0.1:36775)
0808 18:32:47.454827 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:47.454848 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:47.454887 (+ 39us) server_negotiation.cc:407] Connection header received
0808 18:32:47.455704 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:47.455724 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:47.456090 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:47.456470 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:47.456896 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:47.457971 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:47.459051 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:47.459578 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:47.459786 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:47.460886 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:47.460895 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:47.460915 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:47.461361 (+ 446us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:47.461418 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:47.461443 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:47.461605 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:47.461810 (+ 205us) server_negotiation.cc:299] Negotiation successful
0808 18:32:47.461964 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":140,"threads_started":1}
I20260808 18:32:47.463168 20148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:47.464035 20148 log.cc:824] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:47.466287 20148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad: No bootstrap required, opened a new log
I20260808 18:32:47.466281 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:32:47.470635 20148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db59fb1bab4a1bb3ca46057d605bad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260808 18:32:47.470899 20148 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:47.470956 20148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71db59fb1bab4a1bb3ca46057d605bad, State: Initialized, Role: FOLLOWER
I20260808 18:32:47.471421 20148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db59fb1bab4a1bb3ca46057d605bad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260808 18:32:47.471593 20148 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:32:47.471704 20148 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:32:47.471791 20148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:47.472702 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:47.468431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42297 (local address 127.0.0.1:36775)
0808 18:32:47.468583 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:47.468587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:47.468682 (+ 95us) server_negotiation.cc:407] Connection header received
0808 18:32:47.468749 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:47.468752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:47.468795 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:47.468899 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:47.469148 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:47.469931 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:47.470907 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:47.471265 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:47.471369 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:47.472226 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:47.472230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:47.472232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:47.472245 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:47.472272 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:47.472285 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:47.472372 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:47.472517 (+ 145us) server_negotiation.cc:299] Negotiation successful
0808 18:32:47.472555 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 18:32:47.472987 20148 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db59fb1bab4a1bb3ca46057d605bad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260808 18:32:47.473279 20148 leader_election.cc:302] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71db59fb1bab4a1bb3ca46057d605bad; no voters:
I20260808 18:32:47.473556 20148 leader_election.cc:288] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:32:47.473636 20162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:32:47.473814 20162 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad [term 1 LEADER]: Becoming Leader. State: Replica: 71db59fb1bab4a1bb3ca46057d605bad, State: Running, Role: LEADER
I20260808 18:32:47.474119 20162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db59fb1bab4a1bb3ca46057d605bad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260808 18:32:47.475881 20148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:32:47.476042 20164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71db59fb1bab4a1bb3ca46057d605bad. Latest consensus state: current_term: 1 leader_uuid: "71db59fb1bab4a1bb3ca46057d605bad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db59fb1bab4a1bb3ca46057d605bad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } } }
I20260808 18:32:47.476182 20164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:47.476581 20167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:32:47.476543 20163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71db59fb1bab4a1bb3ca46057d605bad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db59fb1bab4a1bb3ca46057d605bad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } } }
I20260808 18:32:47.476689 20163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:47.479791 20167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:32:47.486268 20167 catalog_manager.cc:1347] Generated new cluster ID: e265f5faf356414b8facab81320f607e
I20260808 18:32:47.486356 20167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:32:47.493669 20167 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:32:47.494644 20167 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:32:47.503372 20167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71db59fb1bab4a1bb3ca46057d605bad: Generated new TSK 0
I20260808 18:32:47.504029 20167 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:32:47.512916 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 20070
W20260808 18:32:47.521099 20053 connection.cc:535] client connection to 127.0.0.1:36775 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36775): Connection reset by peer (error 104)
2026-08-08T18:32:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1503 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:32:47 dist-test-slave-gp6n krb5kdc[20184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:32:47 dist-test-slave-gp6n krb5kdc[20184](info): set up 2 sockets
Aug 08 18:32:47 dist-test-slave-gp6n krb5kdc[20184](info): commencing operation
krb5kdc: starting...
W20260808 18:32:48.670130 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:32:48 dist-test-slave-gp6n krb5kdc[20184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:32:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:32:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:32:48.814276 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45471
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:48.925904 20198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:48.926240 20198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:48.926318 20198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:48.930424 20198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:32:48.930540 20198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:48.930588 20198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:48.930648 20198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:32:48.930688 20198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:32:48.936093 20198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45471
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:48.937156 20198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:48.938194 20198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:48.946798 20204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:48.946970 20198 server_base.cc:1034] running on GCE node
W20260808 18:32:48.946805 20205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:48.946820 20207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:48.947808 20198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:48.948927 20198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:48.950135 20198 hybrid_clock.cc:648] HybridClock initialized: now 1786213968950116 us; error 31 us; skew 500 ppm
Aug 08 18:32:48 dist-test-slave-gp6n krb5kdc[20184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:48.952932 20198 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:32:48.953984 20198 webserver.cc:460] Webserver started at http://127.0.0.1:36275/ using document root <none> and password file <none>
I20260808 18:32:48.954697 20198 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:48.954772 20198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:48.954958 20198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:48.957218 20198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "09bb564e8f034ac9986776c9b74a8890"
format_stamp: "Formatted at 2026-08-08 18:32:48 on dist-test-slave-gp6n"
server_key: "357293f8830e4aa5a6871beb41f4e072"
server_key_iv: "3aea89333fb955fb2895ea6f20c61e60"
server_key_version: "encryptionkey@0"
I20260808 18:32:48.957789 20198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "09bb564e8f034ac9986776c9b74a8890"
format_stamp: "Formatted at 2026-08-08 18:32:48 on dist-test-slave-gp6n"
server_key: "357293f8830e4aa5a6871beb41f4e072"
server_key_iv: "3aea89333fb955fb2895ea6f20c61e60"
server_key_version: "encryptionkey@0"
I20260808 18:32:48.961807 20198 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 18:32:48.964382 20214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:48.965664 20198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:32:48.965770 20198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "09bb564e8f034ac9986776c9b74a8890"
format_stamp: "Formatted at 2026-08-08 18:32:48 on dist-test-slave-gp6n"
server_key: "357293f8830e4aa5a6871beb41f4e072"
server_key_iv: "3aea89333fb955fb2895ea6f20c61e60"
server_key_version: "encryptionkey@0"
I20260808 18:32:48.965847 20198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:48.970562 20198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:48.973726 20198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:48.973944 20198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:48.981220 20198 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45471
I20260808 18:32:48.981252 20273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45471 every 8 connection(s)
I20260808 18:32:48.982147 20198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:32:48.985086 20274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:48.990537 20274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890: Bootstrap starting.
I20260808 18:32:48.991158 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 20198
I20260808 18:32:48.991292 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:32:48.991596 30010 external_mini_cluster.cc:1428] Setting key 1f58b9d2a924608f8cad31c16bdeca58
I20260808 18:32:48.993427 20274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:48.994163 20274 log.cc:824] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:48.996542 20274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890: No bootstrap required, opened a new log
I20260808 18:32:48.997193 20277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:48.992906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59985 (local address 127.0.0.1:45471)
0808 18:32:48.993381 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:48.993395 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:48.993432 (+ 37us) server_negotiation.cc:407] Connection header received
0808 18:32:48.994133 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:48.994151 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:48.994629 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:48.995023 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:48.995864 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:48.995868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:48.995879 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:48.995889 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:48.995947 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:48.995965 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:48.996050 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:48.996201 (+ 151us) server_negotiation.cc:299] Negotiation successful
0808 18:32:48.996376 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":156,"threads_started":1}
I20260808 18:32:49.000334 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:32:49.001579 20274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bb564e8f034ac9986776c9b74a8890" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45471 } }
I20260808 18:32:49.001888 20274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:49.001953 20274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09bb564e8f034ac9986776c9b74a8890, State: Initialized, Role: FOLLOWER
I20260808 18:32:49.002441 20274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bb564e8f034ac9986776c9b74a8890" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45471 } }
I20260808 18:32:49.002625 20274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:32:49.002700 20274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:32:49.002810 20274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:49.004135 20274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bb564e8f034ac9986776c9b74a8890" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45471 } }
I20260808 18:32:49.004546 20274 leader_election.cc:302] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09bb564e8f034ac9986776c9b74a8890; no voters:
I20260808 18:32:49.004693 20277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:49.002720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45241 (local address 127.0.0.1:45471)
0808 18:32:49.002822 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:49.002825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:49.003266 (+ 441us) server_negotiation.cc:407] Connection header received
0808 18:32:49.003366 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:49.003368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:49.003398 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:49.003500 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:49.004215 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:49.004218 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:49.004220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:49.004230 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:49.004283 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:49.004298 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:49.004384 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:49.004493 (+ 109us) server_negotiation.cc:299] Negotiation successful
0808 18:32:49.004526 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 18:32:49.004894 20274 leader_election.cc:288] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:32:49.004984 20284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:32:49.005223 20284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890 [term 1 LEADER]: Becoming Leader. State: Replica: 09bb564e8f034ac9986776c9b74a8890, State: Running, Role: LEADER
I20260808 18:32:49.005532 20284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bb564e8f034ac9986776c9b74a8890" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45471 } }
I20260808 18:32:49.006172 20274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:32:49.006801 20287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09bb564e8f034ac9986776c9b74a8890. Latest consensus state: current_term: 1 leader_uuid: "09bb564e8f034ac9986776c9b74a8890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bb564e8f034ac9986776c9b74a8890" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45471 } } }
I20260808 18:32:49.006945 20287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890 [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:49.007309 20286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09bb564e8f034ac9986776c9b74a8890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bb564e8f034ac9986776c9b74a8890" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45471 } } }
I20260808 18:32:49.007411 20286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890 [sys.catalog]: This master's current role is: LEADER
W20260808 18:32:49.011493 20303 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:32:49.011574 20303 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:32:49.011655 20293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:32:49.012394 20293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:32:49.017776 20293 catalog_manager.cc:1347] Generated new cluster ID: 803ba219339047b4835f696f76efa9f2
I20260808 18:32:49.017872 20293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:32:49.038537 20293 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:32:49.038882 20293 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:32:49.047010 20293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09bb564e8f034ac9986776c9b74a8890: Generated new TSK 0
I20260808 18:32:49.047736 20293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:32:49.054396 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 20198
2026-08-08T18:32:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1541 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:32:49 dist-test-slave-gp6n krb5kdc[20310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:32:49 dist-test-slave-gp6n krb5kdc[20310](info): set up 2 sockets
Aug 08 18:32:49 dist-test-slave-gp6n krb5kdc[20310](info): commencing operation
krb5kdc: starting...
W20260808 18:32:50.267800 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.191s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:32:50 dist-test-slave-gp6n krb5kdc[20310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:32:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:32:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:32:50.399317 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33181
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:50.511217 20325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:50.511509 20325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:50.511581 20325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:50.515534 20325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:32:50.515632 20325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:50.515672 20325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:50.515709 20325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:32:50.515745 20325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:32:50.520809 20325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:50.521839 20325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:50.522872 20325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:50.531373 20330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:50.531368 20333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:50.531459 20331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:50.531663 20325 server_base.cc:1034] running on GCE node
I20260808 18:32:50.532182 20325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:50.533221 20325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:50.534379 20325 hybrid_clock.cc:648] HybridClock initialized: now 1786213970534361 us; error 30 us; skew 500 ppm
Aug 08 18:32:50 dist-test-slave-gp6n krb5kdc[20310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:50.537196 20325 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:32:50.538077 20325 webserver.cc:460] Webserver started at http://127.0.0.1:39869/ using document root <none> and password file <none>
I20260808 18:32:50.538678 20325 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:50.538771 20325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:50.539005 20325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:50.540948 20325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "2c3195acaba9484caa39d9cc3029e8be"
format_stamp: "Formatted at 2026-08-08 18:32:50 on dist-test-slave-gp6n"
server_key: "97ca04b3ffe4a7b3e75e8e2a404a9078"
server_key_iv: "e8650d1374368a73da05905398d1520b"
server_key_version: "encryptionkey@0"
I20260808 18:32:50.541551 20325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "2c3195acaba9484caa39d9cc3029e8be"
format_stamp: "Formatted at 2026-08-08 18:32:50 on dist-test-slave-gp6n"
server_key: "97ca04b3ffe4a7b3e75e8e2a404a9078"
server_key_iv: "e8650d1374368a73da05905398d1520b"
server_key_version: "encryptionkey@0"
I20260808 18:32:50.545214 20325 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 18:32:50.547719 20340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:50.548959 20325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:32:50.549124 20325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "2c3195acaba9484caa39d9cc3029e8be"
format_stamp: "Formatted at 2026-08-08 18:32:50 on dist-test-slave-gp6n"
server_key: "97ca04b3ffe4a7b3e75e8e2a404a9078"
server_key_iv: "e8650d1374368a73da05905398d1520b"
server_key_version: "encryptionkey@0"
I20260808 18:32:50.549239 20325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:50.573107 20325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:50.576159 20325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:50.576368 20325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:50.583810 20325 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33181
I20260808 18:32:50.583841 20393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33181 every 8 connection(s)
I20260808 18:32:50.584733 20325 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:32:50.586087 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 20325
I20260808 18:32:50.586232 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:32:50.586613 30010 external_mini_cluster.cc:1428] Setting key bde02e99d5ce8d99cd74a4006a60ba52
I20260808 18:32:50.587914 20394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:50.594729 20394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be: Bootstrap starting.
Aug 08 18:32:50 dist-test-slave-gp6n krb5kdc[20310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 18:32:50.597280 20394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:50.597985 20394 log.cc:824] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:50.599864 20394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be: No bootstrap required, opened a new log
I20260808 18:32:50.601763 20398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:50.588087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50725 (local address 127.0.0.1:33181)
0808 18:32:50.588502 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:50.588522 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:50.588565 (+ 43us) server_negotiation.cc:407] Connection header received
0808 18:32:50.589440 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:50.589463 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:50.589888 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:50.590249 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:50.590996 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:50.592487 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:50.593336 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:50.594053 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:50.594210 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:50.596877 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:50.596882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:50.596897 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:50.596927 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:50.599008 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:50.599466 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:50.599472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:50.599479 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:50.599599 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:50.599899 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:50.599902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:50.599904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:50.600328 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:50.600554 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:50.600819 (+ 265us) server_negotiation.cc:299] Negotiation successful
0808 18:32:50.601091 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":115,"threads_started":1}
I20260808 18:32:50.603401 20394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3195acaba9484caa39d9cc3029e8be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33181 } }
I20260808 18:32:50.603711 20394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:50.603798 20394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3195acaba9484caa39d9cc3029e8be, State: Initialized, Role: FOLLOWER
I20260808 18:32:50.604225 20394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3195acaba9484caa39d9cc3029e8be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33181 } }
I20260808 18:32:50.604348 20394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:32:50.604435 20394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:32:50.604545 20394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:50.604568 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:32:50.605638 20394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3195acaba9484caa39d9cc3029e8be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33181 } }
I20260808 18:32:50.606096 20394 leader_election.cc:302] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c3195acaba9484caa39d9cc3029e8be; no voters:
I20260808 18:32:50.606443 20394 leader_election.cc:288] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:32:50.606583 20405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:32:50.606814 20405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be [term 1 LEADER]: Becoming Leader. State: Replica: 2c3195acaba9484caa39d9cc3029e8be, State: Running, Role: LEADER
I20260808 18:32:50.607132 20405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3195acaba9484caa39d9cc3029e8be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33181 } }
I20260808 18:32:50.607687 20394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:32:50.608237 20405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c3195acaba9484caa39d9cc3029e8be. Latest consensus state: current_term: 1 leader_uuid: "2c3195acaba9484caa39d9cc3029e8be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3195acaba9484caa39d9cc3029e8be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33181 } } }
I20260808 18:32:50.608361 20405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:50.608744 20406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c3195acaba9484caa39d9cc3029e8be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3195acaba9484caa39d9cc3029e8be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33181 } } }
I20260808 18:32:50.608860 20406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be [sys.catalog]: This master's current role is: LEADER
W20260808 18:32:50.612747 20426 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:32:50.612830 20426 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:32:50.612890 20412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:32:50.613749 20412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:32:50.617789 20398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:50.607210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46197 (local address 127.0.0.1:33181)
0808 18:32:50.607317 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:50.607321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:50.610393 (+ 3072us) server_negotiation.cc:407] Connection header received
0808 18:32:50.610571 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:50.610574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:50.610622 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:50.610729 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:50.611378 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:50.611909 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:50.612747 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:50.613065 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:50.613182 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:50.614633 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:50.614637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:50.614640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:50.614664 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:50.616131 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:50.616572 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:50.616575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:50.616577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:50.616657 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:50.616946 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:50.616950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:50.616952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:50.617125 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:50.617269 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:50.617514 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 18:32:50.617633 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 18:32:50.619585 20412 catalog_manager.cc:1347] Generated new cluster ID: d0143e74c6d7493cacd40fd1bdbd1612
I20260808 18:32:50.619642 20412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:32:50.635528 20412 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:32:50.636686 20412 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:32:50.646031 20412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c3195acaba9484caa39d9cc3029e8be: Generated new TSK 0
I20260808 18:32:50.646903 20412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:32:50.656829 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 20325
2026-08-08T18:32:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1602 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:32:50 dist-test-slave-gp6n krb5kdc[20434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:32:50 dist-test-slave-gp6n krb5kdc[20434](info): set up 2 sockets
Aug 08 18:32:50 dist-test-slave-gp6n krb5kdc[20434](info): commencing operation
krb5kdc: starting...
W20260808 18:32:51.831007 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:32:51 dist-test-slave-gp6n krb5kdc[20434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:32:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:32:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:32:51.962793 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43411
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:52.074438 20448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:52.074770 20448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:52.074853 20448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:52.078799 20448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:32:52.078907 20448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:52.078949 20448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:52.078986 20448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:32:52.079021 20448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:32:52.084172 20448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:52.085261 20448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:52.086308 20448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:52.094990 20453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:52.094967 20456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:52.094969 20454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:52.095129 20448 server_base.cc:1034] running on GCE node
I20260808 18:32:52.095974 20448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:52.097150 20448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:52.098362 20448 hybrid_clock.cc:648] HybridClock initialized: now 1786213972098329 us; error 44 us; skew 500 ppm
Aug 08 18:32:52 dist-test-slave-gp6n krb5kdc[20434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:52.101125 20448 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:32:52.102038 20448 webserver.cc:460] Webserver started at http://127.0.0.1:38587/ using document root <none> and password file <none>
I20260808 18:32:52.102663 20448 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:52.102761 20448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:52.103008 20448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:52.105072 20448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "1adc7c4c3fb049939fd2fbc450c51946"
format_stamp: "Formatted at 2026-08-08 18:32:52 on dist-test-slave-gp6n"
server_key: "4e9d2209682288c15a34538b641e667c"
server_key_iv: "f9c2d49bbd9d8af4c7c61fcf6ead5567"
server_key_version: "encryptionkey@0"
I20260808 18:32:52.105664 20448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "1adc7c4c3fb049939fd2fbc450c51946"
format_stamp: "Formatted at 2026-08-08 18:32:52 on dist-test-slave-gp6n"
server_key: "4e9d2209682288c15a34538b641e667c"
server_key_iv: "f9c2d49bbd9d8af4c7c61fcf6ead5567"
server_key_version: "encryptionkey@0"
I20260808 18:32:52.109529 20448 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 18:32:52.112104 20463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:52.113384 20448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:32:52.113533 20448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "1adc7c4c3fb049939fd2fbc450c51946"
format_stamp: "Formatted at 2026-08-08 18:32:52 on dist-test-slave-gp6n"
server_key: "4e9d2209682288c15a34538b641e667c"
server_key_iv: "f9c2d49bbd9d8af4c7c61fcf6ead5567"
server_key_version: "encryptionkey@0"
I20260808 18:32:52.113651 20448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:52.138959 20448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:52.142046 20448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:52.142266 20448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:52.149950 20448 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43411
I20260808 18:32:52.149977 20522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43411 every 8 connection(s)
I20260808 18:32:52.151047 20448 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:32:52.153883 20523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:52.159395 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 20448
I20260808 18:32:52.159549 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:32:52.159874 30010 external_mini_cluster.cc:1428] Setting key 64b708234208a2eb701e79a14e344c56
I20260808 18:32:52.160027 20523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946: Bootstrap starting.
I20260808 18:32:52.162886 20523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:52.163805 20523 log.cc:824] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:52.166128 20523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946: No bootstrap required, opened a new log
I20260808 18:32:52.170845 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:52.161323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45997 (local address 127.0.0.1:43411)
0808 18:32:52.161831 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:52.161850 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:52.161894 (+ 44us) server_negotiation.cc:407] Connection header received
0808 18:32:52.162766 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:52.162791 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:52.163272 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:52.163650 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:52.164111 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:52.165491 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:52.166706 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:52.167405 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:52.167568 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:52.168495 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:52.168501 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:52.168518 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:52.169121 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:52.169208 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:52.169245 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:52.169380 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:52.169596 (+ 216us) server_negotiation.cc:299] Negotiation successful
0808 18:32:52.169826 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":139,"threads_started":1}
I20260808 18:32:52.170758 20523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc7c4c3fb049939fd2fbc450c51946" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43411 } }
I20260808 18:32:52.171037 20523 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:52.171149 20523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adc7c4c3fb049939fd2fbc450c51946, State: Initialized, Role: FOLLOWER
I20260808 18:32:52.171665 20523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc7c4c3fb049939fd2fbc450c51946" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43411 } }
I20260808 18:32:52.171828 20523 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:32:52.171901 20523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:32:52.172027 20523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:52.173136 20523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc7c4c3fb049939fd2fbc450c51946" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43411 } }
I20260808 18:32:52.173518 20523 leader_election.cc:302] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1adc7c4c3fb049939fd2fbc450c51946; no voters:
I20260808 18:32:52.173868 20523 leader_election.cc:288] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:32:52.173970 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:32:52.174000 20528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:32:52.174206 20528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946 [term 1 LEADER]: Becoming Leader. State: Replica: 1adc7c4c3fb049939fd2fbc450c51946, State: Running, Role: LEADER
I20260808 18:32:52.174621 20528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc7c4c3fb049939fd2fbc450c51946" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43411 } }
I20260808 18:32:52.175182 20523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:32:52.175921 20528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1adc7c4c3fb049939fd2fbc450c51946. Latest consensus state: current_term: 1 leader_uuid: "1adc7c4c3fb049939fd2fbc450c51946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc7c4c3fb049939fd2fbc450c51946" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43411 } } }
I20260808 18:32:52.176119 20528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946 [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:52.176379 20529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1adc7c4c3fb049939fd2fbc450c51946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adc7c4c3fb049939fd2fbc450c51946" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43411 } } }
I20260808 18:32:52.176501 20529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946 [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:52.176575 20541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:32:52.179886 20541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:32:52.181671 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:52.176342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46401 (local address 127.0.0.1:43411)
0808 18:32:52.176635 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:52.176639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:52.177324 (+ 685us) server_negotiation.cc:407] Connection header received
0808 18:32:52.177518 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:52.177522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:52.177571 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:52.177660 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:52.177946 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:52.178714 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:52.179616 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:52.179930 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:52.180040 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:52.180959 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:52.180963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:52.180968 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:52.180981 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:52.181010 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:52.181023 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:52.181083 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:52.181488 (+ 405us) server_negotiation.cc:299] Negotiation successful
0808 18:32:52.181543 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260808 18:32:52.186095 20541 catalog_manager.cc:1347] Generated new cluster ID: 8a81b3f80183459581d064ef7c5d71e8
I20260808 18:32:52.186165 20541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:32:52.218890 20541 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:32:52.220605 20541 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:32:52.227396 20541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1adc7c4c3fb049939fd2fbc450c51946: Generated new TSK 0
I20260808 18:32:52.228068 20541 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:32:52.237706 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 20448
2026-08-08T18:32:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1581 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:32:52 dist-test-slave-gp6n krb5kdc[20556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:32:52 dist-test-slave-gp6n krb5kdc[20556](info): set up 2 sockets
Aug 08 18:32:52 dist-test-slave-gp6n krb5kdc[20556](info): commencing operation
krb5kdc: starting...
W20260808 18:32:53.347151 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:32:53 dist-test-slave-gp6n krb5kdc[20556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:32:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:32:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:32:53.482991 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34911
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:53.592113 20570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:53.592489 20570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:53.592581 20570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:53.596570 20570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:32:53.596671 20570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:53.596709 20570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:53.596747 20570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:32:53.596777 20570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:32:53.601807 20570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:53.602815 20570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:53.603816 20570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:53.612146 20575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:53.612146 20576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:53.612239 20578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:53.612325 20570 server_base.cc:1034] running on GCE node
I20260808 18:32:53.613106 20570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:53.614167 20570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:53.615393 20570 hybrid_clock.cc:648] HybridClock initialized: now 1786213973615384 us; error 51 us; skew 500 ppm
Aug 08 18:32:53 dist-test-slave-gp6n krb5kdc[20556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:53.623534 20570 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:32:53.624493 20570 webserver.cc:460] Webserver started at http://127.0.0.1:44111/ using document root <none> and password file <none>
I20260808 18:32:53.625021 20570 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:53.625083 20570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:53.625252 20570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:53.627395 20570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "0330b5e8d2044329bf98364a5146d3b0"
format_stamp: "Formatted at 2026-08-08 18:32:53 on dist-test-slave-gp6n"
server_key: "7272127dcff48b8dfdad807f5af99d9d"
server_key_iv: "68b0f2dca31c810df4029aee733a1a0b"
server_key_version: "encryptionkey@0"
I20260808 18:32:53.627929 20570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "0330b5e8d2044329bf98364a5146d3b0"
format_stamp: "Formatted at 2026-08-08 18:32:53 on dist-test-slave-gp6n"
server_key: "7272127dcff48b8dfdad807f5af99d9d"
server_key_iv: "68b0f2dca31c810df4029aee733a1a0b"
server_key_version: "encryptionkey@0"
I20260808 18:32:53.631750 20570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:32:53.634078 20585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:53.635413 20570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:32:53.635540 20570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "0330b5e8d2044329bf98364a5146d3b0"
format_stamp: "Formatted at 2026-08-08 18:32:53 on dist-test-slave-gp6n"
server_key: "7272127dcff48b8dfdad807f5af99d9d"
server_key_iv: "68b0f2dca31c810df4029aee733a1a0b"
server_key_version: "encryptionkey@0"
I20260808 18:32:53.635624 20570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:53.640121 20570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:53.643134 20570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:53.643275 20570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:53.650264 20570 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34911
I20260808 18:32:53.650278 20643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34911 every 8 connection(s)
I20260808 18:32:53.651225 20570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:32:53.653891 20644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:53.659161 20644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0: Bootstrap starting.
I20260808 18:32:53.659688 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 20570
I20260808 18:32:53.659850 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:32:53.660140 30010 external_mini_cluster.cc:1428] Setting key 58583857e5dea1a7d787aa5570d3b7b7
I20260808 18:32:53.661777 20644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:53.662580 20644 log.cc:824] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:53.664772 20644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0: No bootstrap required, opened a new log
I20260808 18:32:53.665424 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:53.661388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59887 (local address 127.0.0.1:34911)
0808 18:32:53.661855 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:53.661875 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:53.661923 (+ 48us) server_negotiation.cc:407] Connection header received
0808 18:32:53.662604 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:53.662622 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:53.662941 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:53.663211 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:53.664004 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:53.664011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:53.664027 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:53.664039 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:53.664101 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:53.664126 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:53.664239 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:53.664408 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 18:32:53.664554 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":149,"threads_started":1}
I20260808 18:32:53.668256 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:32:53.668862 20644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0330b5e8d2044329bf98364a5146d3b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34911 } }
I20260808 18:32:53.669059 20644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:53.669091 20644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0330b5e8d2044329bf98364a5146d3b0, State: Initialized, Role: FOLLOWER
I20260808 18:32:53.669603 20644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0330b5e8d2044329bf98364a5146d3b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34911 } }
I20260808 18:32:53.669770 20644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:32:53.669838 20644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:32:53.669931 20644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:53.671082 20644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0330b5e8d2044329bf98364a5146d3b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34911 } }
I20260808 18:32:53.671415 20644 leader_election.cc:302] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0330b5e8d2044329bf98364a5146d3b0; no voters:
I20260808 18:32:53.671757 20644 leader_election.cc:288] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:32:53.671892 20656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:32:53.672114 20656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0 [term 1 LEADER]: Becoming Leader. State: Replica: 0330b5e8d2044329bf98364a5146d3b0, State: Running, Role: LEADER
I20260808 18:32:53.672245 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:53.670513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59263 (local address 127.0.0.1:34911)
0808 18:32:53.670634 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:53.670639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:53.670795 (+ 156us) server_negotiation.cc:407] Connection header received
0808 18:32:53.670883 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:53.670888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:53.670926 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:53.671020 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:53.671748 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:53.671752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:53.671754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:53.671768 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:53.671798 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:53.671814 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:53.671896 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:53.672080 (+ 184us) server_negotiation.cc:299] Negotiation successful
0808 18:32:53.672123 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 18:32:53.672489 20656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0330b5e8d2044329bf98364a5146d3b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34911 } }
I20260808 18:32:53.673009 20644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:32:53.673686 20656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0330b5e8d2044329bf98364a5146d3b0. Latest consensus state: current_term: 1 leader_uuid: "0330b5e8d2044329bf98364a5146d3b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0330b5e8d2044329bf98364a5146d3b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34911 } } }
I20260808 18:32:53.673880 20656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0 [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:53.674135 20657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0330b5e8d2044329bf98364a5146d3b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0330b5e8d2044329bf98364a5146d3b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34911 } } }
I20260808 18:32:53.674264 20657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0 [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:53.674448 20665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:32:53.677677 20665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:32:53.683192 20665 catalog_manager.cc:1347] Generated new cluster ID: 75ce8929ab8d4f7f9e3a7b5b30a75ee1
I20260808 18:32:53.683257 20665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:32:53.704499 20665 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:32:53.704807 20665 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:32:53.711081 20665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0330b5e8d2044329bf98364a5146d3b0: Generated new TSK 0
I20260808 18:32:53.711789 20665 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:32:53.727044 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 20570
2026-08-08T18:32:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1489 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:32:53 dist-test-slave-gp6n krb5kdc[20680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:32:53 dist-test-slave-gp6n krb5kdc[20680](info): set up 2 sockets
Aug 08 18:32:53 dist-test-slave-gp6n krb5kdc[20680](info): commencing operation
krb5kdc: starting...
W20260808 18:32:54.820400 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:32:54 dist-test-slave-gp6n krb5kdc[20680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:32:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:32:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:32:54.954154 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33333
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:55.067566 20694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:55.067852 20694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:55.067896 20694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:55.072023 20694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:32:55.072088 20694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:55.072108 20694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:55.072125 20694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:32:55.072140 20694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:32:55.077586 20694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:55.078946 20694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:55.080104 20694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:55.088773 20700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:55.088801 20699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:55.088802 20702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:55.089145 20694 server_base.cc:1034] running on GCE node
I20260808 18:32:55.089675 20694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:55.090890 20694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:55.092070 20694 hybrid_clock.cc:648] HybridClock initialized: now 1786213975092035 us; error 49 us; skew 500 ppm
Aug 08 18:32:55 dist-test-slave-gp6n krb5kdc[20680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:55.095016 20694 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:32:55.095902 20694 webserver.cc:460] Webserver started at http://127.0.0.1:36987/ using document root <none> and password file <none>
I20260808 18:32:55.096494 20694 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:55.096556 20694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:55.096721 20694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:55.098892 20694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "82d4746cd9e4416aa4c9f83c6ef30a77"
format_stamp: "Formatted at 2026-08-08 18:32:55 on dist-test-slave-gp6n"
server_key: "23064365f1ad5e69fb285f56112bfd9a"
server_key_iv: "93e9a086957da1df783e135b3578de30"
server_key_version: "encryptionkey@0"
I20260808 18:32:55.099475 20694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "82d4746cd9e4416aa4c9f83c6ef30a77"
format_stamp: "Formatted at 2026-08-08 18:32:55 on dist-test-slave-gp6n"
server_key: "23064365f1ad5e69fb285f56112bfd9a"
server_key_iv: "93e9a086957da1df783e135b3578de30"
server_key_version: "encryptionkey@0"
I20260808 18:32:55.103250 20694 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:32:55.105695 20710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:55.107136 20694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:32:55.107263 20694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "82d4746cd9e4416aa4c9f83c6ef30a77"
format_stamp: "Formatted at 2026-08-08 18:32:55 on dist-test-slave-gp6n"
server_key: "23064365f1ad5e69fb285f56112bfd9a"
server_key_iv: "93e9a086957da1df783e135b3578de30"
server_key_version: "encryptionkey@0"
I20260808 18:32:55.107374 20694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:55.132558 20694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:55.136073 20694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:55.136309 20694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:55.144493 20694 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33333
I20260808 18:32:55.144526 20768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33333 every 8 connection(s)
I20260808 18:32:55.145532 20694 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:32:55.148365 20769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:55.150918 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 20694
I20260808 18:32:55.151093 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:32:55.151412 30010 external_mini_cluster.cc:1428] Setting key 092c694fdb877443d102757c3b01d7b0
I20260808 18:32:55.154609 20769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77: Bootstrap starting.
I20260808 18:32:55.157308 20769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:55.158064 20769 log.cc:824] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:55.160230 20769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77: No bootstrap required, opened a new log
Aug 08 18:32:55 dist-test-slave-gp6n krb5kdc[20680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 18:32:55.164472 20769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d4746cd9e4416aa4c9f83c6ef30a77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33333 } }
I20260808 18:32:55.164763 20769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:55.164822 20769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d4746cd9e4416aa4c9f83c6ef30a77, State: Initialized, Role: FOLLOWER
I20260808 18:32:55.165313 20769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d4746cd9e4416aa4c9f83c6ef30a77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33333 } }
I20260808 18:32:55.165444 20769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:32:55.165529 20769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:32:55.165635 20769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:55.166622 20769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d4746cd9e4416aa4c9f83c6ef30a77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33333 } }
I20260808 18:32:55.167034 20769 leader_election.cc:302] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82d4746cd9e4416aa4c9f83c6ef30a77; no voters:
I20260808 18:32:55.167448 20769 leader_election.cc:288] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:32:55.167544 20775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:32:55.167827 20775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77 [term 1 LEADER]: Becoming Leader. State: Replica: 82d4746cd9e4416aa4c9f83c6ef30a77, State: Running, Role: LEADER
I20260808 18:32:55.168211 20775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d4746cd9e4416aa4c9f83c6ef30a77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33333 } }
I20260808 18:32:55.168716 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:55.152901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:56885 (local address 127.0.0.1:33333)
0808 18:32:55.153376 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:55.153396 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:55.153441 (+ 45us) server_negotiation.cc:407] Connection header received
0808 18:32:55.154236 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:55.154259 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:55.154732 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:55.155088 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:55.155860 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:55.157348 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:55.158338 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:55.158955 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:55.159133 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:55.162006 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:55.162014 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:55.162030 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:55.162060 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:55.164835 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:55.165638 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:55.165644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:55.165651 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:55.165845 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:55.166236 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:55.166239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:55.166241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:55.166782 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:55.167000 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:55.167407 (+ 407us) server_negotiation.cc:299] Negotiation successful
0808 18:32:55.167669 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":124,"threads_started":1}
I20260808 18:32:55.168862 20769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:32:55.169977 20777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82d4746cd9e4416aa4c9f83c6ef30a77. Latest consensus state: current_term: 1 leader_uuid: "82d4746cd9e4416aa4c9f83c6ef30a77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d4746cd9e4416aa4c9f83c6ef30a77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33333 } } }
I20260808 18:32:55.170100 20777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77 [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:55.170459 20776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82d4746cd9e4416aa4c9f83c6ef30a77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d4746cd9e4416aa4c9f83c6ef30a77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33333 } } }
I20260808 18:32:55.170537 20776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77 [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:55.172214 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260808 18:32:55.173945 20794 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:32:55.174038 20794 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:32:55.174223 20786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:32:55.175138 20786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:32:55.181318 20786 catalog_manager.cc:1347] Generated new cluster ID: 2d78a796d4f842bf9638f33431652149
I20260808 18:32:55.181388 20786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:32:55.183395 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:55.174455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:55593 (local address 127.0.0.1:33333)
0808 18:32:55.174567 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:55.174571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:55.174836 (+ 265us) server_negotiation.cc:407] Connection header received
0808 18:32:55.174980 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:55.174988 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:55.175028 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:55.175117 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:55.175777 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:55.176306 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:55.177315 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:55.177634 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:55.177707 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:55.179546 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:55.179550 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:55.179553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:55.179579 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:55.181242 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:55.181925 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:55.181929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:55.181931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:55.182022 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:55.182462 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:55.182467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:55.182468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:55.182647 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:55.182841 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:55.183106 (+ 265us) server_negotiation.cc:299] Negotiation successful
0808 18:32:55.183205 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 18:32:55.198225 20786 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:32:55.199146 20786 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:32:55.211959 20786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82d4746cd9e4416aa4c9f83c6ef30a77: Generated new TSK 0
I20260808 18:32:55.212715 20786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:32:55.220831 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 20694
W20260808 18:32:55.230368 20677 connection.cc:535] client connection to 127.0.0.1:33333 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33333): Connection reset by peer (error 104)
2026-08-08T18:32:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1496 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:32:55 dist-test-slave-gp6n krb5kdc[20805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:32:55 dist-test-slave-gp6n krb5kdc[20805](info): set up 2 sockets
Aug 08 18:32:55 dist-test-slave-gp6n krb5kdc[20805](info): commencing operation
krb5kdc: starting...
W20260808 18:32:56.347518 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:32:56 dist-test-slave-gp6n krb5kdc[20805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:32:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:32:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:32:56.477809 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35357
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:56.588217 20821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:56.588505 20821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:56.588548 20821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:56.592504 20821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:32:56.592558 20821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:56.592577 20821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:56.592594 20821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:32:56.592609 20821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:32:56.597612 20821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:56.598628 20821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:56.599586 20821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:32:56.608055 20821 server_base.cc:1034] running on GCE node
W20260808 18:32:56.607985 20829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:56.608016 20826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:56.608003 20827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:56.608866 20821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:56.609944 20821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:56.611104 20821 hybrid_clock.cc:648] HybridClock initialized: now 1786213976611073 us; error 45 us; skew 500 ppm
Aug 08 18:32:56 dist-test-slave-gp6n krb5kdc[20805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:56.614050 20821 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:32:56.615031 20821 webserver.cc:460] Webserver started at http://127.0.0.1:44309/ using document root <none> and password file <none>
I20260808 18:32:56.615584 20821 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:56.615679 20821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:56.615912 20821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:56.617950 20821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "1cc2285fb0444b64a7252bf2c212e77f"
format_stamp: "Formatted at 2026-08-08 18:32:56 on dist-test-slave-gp6n"
server_key: "612a0d421c3ad38e6ddd02a59b146536"
server_key_iv: "dc936ba630e4f72c45a1f1281c2ff3a0"
server_key_version: "encryptionkey@0"
I20260808 18:32:56.618597 20821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "1cc2285fb0444b64a7252bf2c212e77f"
format_stamp: "Formatted at 2026-08-08 18:32:56 on dist-test-slave-gp6n"
server_key: "612a0d421c3ad38e6ddd02a59b146536"
server_key_iv: "dc936ba630e4f72c45a1f1281c2ff3a0"
server_key_version: "encryptionkey@0"
I20260808 18:32:56.622651 20821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:32:56.625241 20836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:56.626658 20821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:32:56.626792 20821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "1cc2285fb0444b64a7252bf2c212e77f"
format_stamp: "Formatted at 2026-08-08 18:32:56 on dist-test-slave-gp6n"
server_key: "612a0d421c3ad38e6ddd02a59b146536"
server_key_iv: "dc936ba630e4f72c45a1f1281c2ff3a0"
server_key_version: "encryptionkey@0"
I20260808 18:32:56.626914 20821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:56.651469 20821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:56.654525 20821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:56.654726 20821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:56.661897 20821 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35357
I20260808 18:32:56.661922 20893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35357 every 8 connection(s)
I20260808 18:32:56.662918 20821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:32:56.664479 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 20821
I20260808 18:32:56.664628 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:32:56.664942 30010 external_mini_cluster.cc:1428] Setting key 4b0027683610f9a447f7288fb13e4f1c
I20260808 18:32:56.665834 20894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:56.671471 20894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f: Bootstrap starting.
I20260808 18:32:56.673703 20894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:56.673861 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:56.666381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:37561 (local address 127.0.0.1:35357)
0808 18:32:56.666866 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:56.666882 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:56.666917 (+ 35us) server_negotiation.cc:407] Connection header received
0808 18:32:56.667537 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:56.667553 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:56.667884 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:56.668212 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:56.668618 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:56.669644 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:56.670649 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:56.671112 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:56.671279 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:56.672250 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:56.672255 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:56.672267 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:56.672637 (+ 370us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:56.672686 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:56.672708 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:56.672858 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:56.673003 (+ 145us) server_negotiation.cc:299] Negotiation successful
0808 18:32:56.673138 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":149,"threads_started":1}
I20260808 18:32:56.674562 20894 log.cc:824] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:56.676657 20894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f: No bootstrap required, opened a new log
I20260808 18:32:56.676891 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:32:56.680802 20894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc2285fb0444b64a7252bf2c212e77f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35357 } }
I20260808 18:32:56.681011 20894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:56.681074 20894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc2285fb0444b64a7252bf2c212e77f, State: Initialized, Role: FOLLOWER
I20260808 18:32:56.681480 20894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc2285fb0444b64a7252bf2c212e77f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35357 } }
I20260808 18:32:56.681631 20894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:32:56.681716 20894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:32:56.681834 20894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:56.682760 20894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc2285fb0444b64a7252bf2c212e77f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35357 } }
I20260808 18:32:56.683084 20894 leader_election.cc:302] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cc2285fb0444b64a7252bf2c212e77f; no voters:
I20260808 18:32:56.683431 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:56.679197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:56039 (local address 127.0.0.1:35357)
0808 18:32:56.679327 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:56.679332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:56.679530 (+ 198us) server_negotiation.cc:407] Connection header received
0808 18:32:56.679657 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:56.679660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:56.679706 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:56.679800 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:56.680077 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:56.680576 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:56.681575 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:56.681972 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:56.682086 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:56.682947 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:56.682950 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:56.682953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:56.682965 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:56.682998 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:56.683008 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:56.683106 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:56.683227 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 18:32:56.683262 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 18:32:56.683434 20894 leader_election.cc:288] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:32:56.683782 20908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:32:56.684069 20908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f [term 1 LEADER]: Becoming Leader. State: Replica: 1cc2285fb0444b64a7252bf2c212e77f, State: Running, Role: LEADER
I20260808 18:32:56.684540 20908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc2285fb0444b64a7252bf2c212e77f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35357 } }
I20260808 18:32:56.684777 20894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:32:56.686084 20909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cc2285fb0444b64a7252bf2c212e77f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc2285fb0444b64a7252bf2c212e77f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35357 } } }
I20260808 18:32:56.686199 20909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:56.686764 20921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:32:56.686868 20910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cc2285fb0444b64a7252bf2c212e77f. Latest consensus state: current_term: 1 leader_uuid: "1cc2285fb0444b64a7252bf2c212e77f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc2285fb0444b64a7252bf2c212e77f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35357 } } }
I20260808 18:32:56.686955 20910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:56.689275 20921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:32:56.695057 20921 catalog_manager.cc:1347] Generated new cluster ID: a5ba87953ad1475193ce9e90a7c092e0
I20260808 18:32:56.695132 20921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:32:56.703871 20921 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:32:56.704856 20921 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:32:56.710995 20921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cc2285fb0444b64a7252bf2c212e77f: Generated new TSK 0
I20260808 18:32:56.711565 20921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:32:56.717854 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 20821
W20260808 18:32:56.729578 20803 connection.cc:535] client connection to 127.0.0.1:35357 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35357): Connection reset by peer (error 104)
2026-08-08T18:32:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1498 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:32:56 dist-test-slave-gp6n krb5kdc[20932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:32:56 dist-test-slave-gp6n krb5kdc[20932](info): set up 2 sockets
Aug 08 18:32:56 dist-test-slave-gp6n krb5kdc[20932](info): commencing operation
krb5kdc: starting...
W20260808 18:32:57.882750 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:32:57 dist-test-slave-gp6n krb5kdc[20932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:32:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:32:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:32:58.034574 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36655
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:58.152248 20945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:58.152565 20945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:58.152649 20945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:58.156657 20945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:32:58.156755 20945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:58.156795 20945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:58.156831 20945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:32:58.156863 20945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:32:58.162215 20945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:58.163372 20945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:58.164448 20945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:58.173266 20950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:58.173264 20951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:58.173286 20953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:58.173628 20945 server_base.cc:1034] running on GCE node
I20260808 18:32:58.174250 20945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:58.175453 20945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:58.176674 20945 hybrid_clock.cc:648] HybridClock initialized: now 1786213978176649 us; error 47 us; skew 500 ppm
Aug 08 18:32:58 dist-test-slave-gp6n krb5kdc[20932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:58.179685 20945 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:32:58.180526 20945 webserver.cc:460] Webserver started at http://127.0.0.1:41849/ using document root <none> and password file <none>
I20260808 18:32:58.181036 20945 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:58.181094 20945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:58.181334 20945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:58.183490 20945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "b87c15c48080485c869f89532a8a5cd5"
format_stamp: "Formatted at 2026-08-08 18:32:58 on dist-test-slave-gp6n"
server_key: "78896b7be56487c52f67c078774ef463"
server_key_iv: "50d5d9904c1a936484655f4dee3dbe0d"
server_key_version: "encryptionkey@0"
I20260808 18:32:58.184039 20945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "b87c15c48080485c869f89532a8a5cd5"
format_stamp: "Formatted at 2026-08-08 18:32:58 on dist-test-slave-gp6n"
server_key: "78896b7be56487c52f67c078774ef463"
server_key_iv: "50d5d9904c1a936484655f4dee3dbe0d"
server_key_version: "encryptionkey@0"
I20260808 18:32:58.187736 20945 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:32:58.190137 20960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:58.191766 20945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:32:58.191874 20945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "b87c15c48080485c869f89532a8a5cd5"
format_stamp: "Formatted at 2026-08-08 18:32:58 on dist-test-slave-gp6n"
server_key: "78896b7be56487c52f67c078774ef463"
server_key_iv: "50d5d9904c1a936484655f4dee3dbe0d"
server_key_version: "encryptionkey@0"
I20260808 18:32:58.191959 20945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:58.196607 20945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:58.199951 20945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:58.200098 20945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:58.207324 20945 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36655
I20260808 18:32:58.207412 21017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36655 every 8 connection(s)
I20260808 18:32:58.208279 20945 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:32:58.211001 21018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:58.211874 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 20945
I20260808 18:32:58.211982 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:32:58.212292 30010 external_mini_cluster.cc:1428] Setting key 52a34151cf4eadef054dea525d64de49
I20260808 18:32:58.217429 21018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5: Bootstrap starting.
I20260808 18:32:58.217960 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:58.213607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:39051 (local address 127.0.0.1:36655)
0808 18:32:58.214047 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:58.214062 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:58.214099 (+ 37us) server_negotiation.cc:407] Connection header received
0808 18:32:58.214806 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:58.214825 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:58.215104 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:58.215385 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:58.216255 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:58.216263 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:58.216284 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:58.216301 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:58.216377 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:58.216409 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:58.216566 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:58.216723 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 18:32:58.216952 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":137,"threads_started":1}
I20260808 18:32:58.220706 21018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:58.221336 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:32:58.221688 21018 log.cc:824] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:58.224380 21018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5: No bootstrap required, opened a new log
I20260808 18:32:58.226722 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:58.224147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:56315 (local address 127.0.0.1:36655)
0808 18:32:58.224274 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:58.224279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:58.225005 (+ 726us) server_negotiation.cc:407] Connection header received
0808 18:32:58.225151 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:58.225155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:58.225196 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:58.225284 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:58.226086 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:58.226090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:58.226094 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:32:58.226108 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:58.226138 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:32:58.226153 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:58.226252 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:58.226563 (+ 311us) server_negotiation.cc:299] Negotiation successful
0808 18:32:58.226607 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 18:32:58.229039 21018 raft_consensus.cc:348] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87c15c48080485c869f89532a8a5cd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36655 } }
I20260808 18:32:58.229254 21018 raft_consensus.cc:374] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:58.229285 21018 raft_consensus.cc:729] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87c15c48080485c869f89532a8a5cd5, State: Initialized, Role: FOLLOWER
I20260808 18:32:58.229717 21018 consensus_queue.cc:260] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87c15c48080485c869f89532a8a5cd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36655 } }
I20260808 18:32:58.229838 21018 raft_consensus.cc:388] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:32:58.229887 21018 raft_consensus.cc:482] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:32:58.229956 21018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:58.230873 21018 raft_consensus.cc:504] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87c15c48080485c869f89532a8a5cd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36655 } }
I20260808 18:32:58.231244 21018 leader_election.cc:302] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b87c15c48080485c869f89532a8a5cd5; no voters:
I20260808 18:32:58.231622 21018 leader_election.cc:288] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:32:58.231801 21029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:32:58.232148 21029 raft_consensus.cc:686] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5 [term 1 LEADER]: Becoming Leader. State: Replica: b87c15c48080485c869f89532a8a5cd5, State: Running, Role: LEADER
I20260808 18:32:58.232497 21029 consensus_queue.cc:237] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87c15c48080485c869f89532a8a5cd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36655 } }
I20260808 18:32:58.233011 21018 sys_catalog.cc:564] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:32:58.233577 21029 sys_catalog.cc:455] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b87c15c48080485c869f89532a8a5cd5. Latest consensus state: current_term: 1 leader_uuid: "b87c15c48080485c869f89532a8a5cd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87c15c48080485c869f89532a8a5cd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36655 } } }
I20260808 18:32:58.233692 21029 sys_catalog.cc:458] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5 [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:58.234086 21036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:32:58.234237 21030 sys_catalog.cc:455] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b87c15c48080485c869f89532a8a5cd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87c15c48080485c869f89532a8a5cd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36655 } } }
I20260808 18:32:58.234439 21030 sys_catalog.cc:458] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5 [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:58.237522 21036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:32:58.243280 21036 catalog_manager.cc:1347] Generated new cluster ID: 586d72320ac541f0a4ab5a1ea58a2603
I20260808 18:32:58.243358 21036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:32:58.266281 21036 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:32:58.266646 21036 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:32:58.272179 21036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b87c15c48080485c869f89532a8a5cd5: Generated new TSK 0
I20260808 18:32:58.272848 21036 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:32:58.279057 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 20945
2026-08-08T18:32:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1556 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:32:58 dist-test-slave-gp6n krb5kdc[21053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:32:58 dist-test-slave-gp6n krb5kdc[21053](info): set up 2 sockets
Aug 08 18:32:58 dist-test-slave-gp6n krb5kdc[21053](info): commencing operation
krb5kdc: starting...
W20260808 18:32:59.378022 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:32:59 dist-test-slave-gp6n krb5kdc[21053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:32:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:32:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:32:59.505971 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:32:59.616657 21068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:32:59.616952 21068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:32:59.617030 21068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:32:59.620980 21068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:32:59.621080 21068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:32:59.621121 21068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:32:59.621176 21068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:32:59.621217 21068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:32:59.626355 21068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44127
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:32:59.627545 21068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:32:59.628575 21068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:32:59.636948 21073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:59.636978 21074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:32:59.636943 21078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:32:59.637142 21068 server_base.cc:1034] running on GCE node
I20260808 18:32:59.637844 21068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:32:59.638969 21068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:32:59.640162 21068 hybrid_clock.cc:648] HybridClock initialized: now 1786213979640148 us; error 25 us; skew 500 ppm
Aug 08 18:32:59 dist-test-slave-gp6n krb5kdc[21053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:32:59.647459 21068 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:32:59.648360 21068 webserver.cc:460] Webserver started at http://127.0.0.1:35013/ using document root <none> and password file <none>
I20260808 18:32:59.648874 21068 fs_manager.cc:359] Metadata directory not provided
I20260808 18:32:59.648932 21068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:32:59.649099 21068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:32:59.651153 21068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "11902440ac294237a2bcf567d6043048"
format_stamp: "Formatted at 2026-08-08 18:32:59 on dist-test-slave-gp6n"
server_key: "e539b8676b76014090bd0f71de40dac4"
server_key_iv: "09c942a1f1412f4287cdc619c1e337cf"
server_key_version: "encryptionkey@0"
I20260808 18:32:59.651665 21068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "11902440ac294237a2bcf567d6043048"
format_stamp: "Formatted at 2026-08-08 18:32:59 on dist-test-slave-gp6n"
server_key: "e539b8676b76014090bd0f71de40dac4"
server_key_iv: "09c942a1f1412f4287cdc619c1e337cf"
server_key_version: "encryptionkey@0"
I20260808 18:32:59.655233 21068 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:32:59.657462 21086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:32:59.658814 21068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:32:59.658937 21068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "11902440ac294237a2bcf567d6043048"
format_stamp: "Formatted at 2026-08-08 18:32:59 on dist-test-slave-gp6n"
server_key: "e539b8676b76014090bd0f71de40dac4"
server_key_iv: "09c942a1f1412f4287cdc619c1e337cf"
server_key_version: "encryptionkey@0"
I20260808 18:32:59.659039 21068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:32:59.674090 21068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:32:59.677335 21068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:32:59.677538 21068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:32:59.684906 21068 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44127
I20260808 18:32:59.684930 21145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44127 every 8 connection(s)
I20260808 18:32:59.685806 21068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:32:59.688385 21146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:32:59.692490 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 21068
I20260808 18:32:59.692624 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:32:59.692910 30010 external_mini_cluster.cc:1428] Setting key cf13924d415c2b6aba97255bf46af0ee
I20260808 18:32:59.694386 21146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048: Bootstrap starting.
I20260808 18:32:59.696566 21146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048: Neither blocks nor log segments found. Creating new log.
I20260808 18:32:59.697294 21146 log.cc:824] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048: Log is configured to *not* fsync() on all Append() calls
I20260808 18:32:59.699144 21146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048: No bootstrap required, opened a new log
Aug 08 18:32:59 dist-test-slave-gp6n krb5kdc[21053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 18:32:59.703296 21146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11902440ac294237a2bcf567d6043048" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44127 } }
I20260808 18:32:59.703513 21146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:32:59.703593 21146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11902440ac294237a2bcf567d6043048, State: Initialized, Role: FOLLOWER
I20260808 18:32:59.704013 21146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11902440ac294237a2bcf567d6043048" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44127 } }
I20260808 18:32:59.704134 21146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:32:59.704242 21146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:32:59.704376 21146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:32:59.705366 21146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11902440ac294237a2bcf567d6043048" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44127 } }
I20260808 18:32:59.705699 21146 leader_election.cc:302] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11902440ac294237a2bcf567d6043048; no voters:
I20260808 18:32:59.706005 21146 leader_election.cc:288] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:32:59.706110 21151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:32:59.706343 21151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048 [term 1 LEADER]: Becoming Leader. State: Replica: 11902440ac294237a2bcf567d6043048, State: Running, Role: LEADER
I20260808 18:32:59.706744 21151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11902440ac294237a2bcf567d6043048" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44127 } }
I20260808 18:32:59.707305 21146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:32:59.708098 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:59.694348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:48573 (local address 127.0.0.1:44127)
0808 18:32:59.694792 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:59.694807 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:59.694842 (+ 35us) server_negotiation.cc:407] Connection header received
0808 18:32:59.695460 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:59.695475 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:59.695804 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:59.696147 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:59.697024 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:59.698073 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:59.699061 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:59.699580 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:59.699745 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:59.702729 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:59.702734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:59.702746 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:59.702772 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:59.704769 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:59.705335 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:59.705339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:59.705343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:59.705473 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:59.705819 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:59.705822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:59.705824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:59.706342 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:59.706645 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:59.706946 (+ 301us) server_negotiation.cc:299] Negotiation successful
0808 18:32:59.707172 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":129,"threads_started":1}
I20260808 18:32:59.708340 21152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11902440ac294237a2bcf567d6043048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11902440ac294237a2bcf567d6043048" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44127 } } }
I20260808 18:32:59.708453 21152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048 [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:59.708859 21160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:32:59.709746 21153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11902440ac294237a2bcf567d6043048. Latest consensus state: current_term: 1 leader_uuid: "11902440ac294237a2bcf567d6043048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11902440ac294237a2bcf567d6043048" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44127 } } }
I20260808 18:32:59.709846 21153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048 [sys.catalog]: This master's current role is: LEADER
I20260808 18:32:59.711575 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:32:59.712428 21160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:32:59.717437 21160 catalog_manager.cc:1347] Generated new cluster ID: 47a3cf24f5ca434bacb5d0402982a4fa
I20260808 18:32:59.717492 21160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:32:59.723686 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:32:59.713738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:60069 (local address 127.0.0.1:44127)
0808 18:32:59.713840 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0808 18:32:59.713845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:32:59.714342 (+ 497us) server_negotiation.cc:407] Connection header received
0808 18:32:59.714563 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:32:59.714567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:32:59.714615 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:32:59.714664 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:32:59.715558 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:59.716363 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:59.717680 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:32:59.718060 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:32:59.718137 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:32:59.719911 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:32:59.719915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:32:59.719918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:32:59.719943 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:32:59.721539 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:59.722142 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:59.722146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:59.722148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:59.722230 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:32:59.722666 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:32:59.722669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:32:59.722671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:32:59.722867 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:32:59.722997 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 18:32:59.723426 (+ 429us) server_negotiation.cc:299] Negotiation successful
0808 18:32:59.723529 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 18:32:59.732280 21160 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:32:59.733083 21160 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:32:59.744310 21160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11902440ac294237a2bcf567d6043048: Generated new TSK 0
I20260808 18:32:59.744982 21160 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:32:59.754666 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 21068
W20260808 18:32:59.760905 21051 connection.cc:535] client connection to 127.0.0.1:44127 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44127): Connection reset by peer (error 104)
2026-08-08T18:32:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1474 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:32:59 dist-test-slave-gp6n krb5kdc[21180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:32:59 dist-test-slave-gp6n krb5kdc[21180](info): set up 2 sockets
Aug 08 18:32:59 dist-test-slave-gp6n krb5kdc[21180](info): commencing operation
krb5kdc: starting...
W20260808 18:33:00.879057 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:33:00 dist-test-slave-gp6n krb5kdc[21180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:33:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:33:01.013340 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35157
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:01.131564 21193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:01.131878 21193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:01.131961 21193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:01.135988 21193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:01.136088 21193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:01.136130 21193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:01.136170 21193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:01.136205 21193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:01.141516 21193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35157
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:01.142680 21193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:01.143728 21193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:01.152597 21202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:01.152628 21204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:01.152737 21193 server_base.cc:1034] running on GCE node
W20260808 18:33:01.152616 21201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:01.153512 21193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:01.154729 21193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:01.155937 21193 hybrid_clock.cc:648] HybridClock initialized: now 1786213981155924 us; error 27 us; skew 500 ppm
Aug 08 18:33:01 dist-test-slave-gp6n krb5kdc[21180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:01.158757 21193 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:33:01.159716 21193 webserver.cc:460] Webserver started at http://127.0.0.1:42865/ using document root <none> and password file <none>
I20260808 18:33:01.160301 21193 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:01.160398 21193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:01.160640 21193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:01.162904 21193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "68c6be3962b6450799cfe1a8eafab282"
format_stamp: "Formatted at 2026-08-08 18:33:01 on dist-test-slave-gp6n"
server_key: "e0a602624a88c1fd306e2e11d37b146f"
server_key_iv: "f95b5b40a4a41e391fca9b4d36526159"
server_key_version: "encryptionkey@0"
I20260808 18:33:01.163507 21193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "68c6be3962b6450799cfe1a8eafab282"
format_stamp: "Formatted at 2026-08-08 18:33:01 on dist-test-slave-gp6n"
server_key: "e0a602624a88c1fd306e2e11d37b146f"
server_key_iv: "f95b5b40a4a41e391fca9b4d36526159"
server_key_version: "encryptionkey@0"
I20260808 18:33:01.167661 21193 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 18:33:01.170697 21212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:01.172081 21193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 18:33:01.172235 21193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "68c6be3962b6450799cfe1a8eafab282"
format_stamp: "Formatted at 2026-08-08 18:33:01 on dist-test-slave-gp6n"
server_key: "e0a602624a88c1fd306e2e11d37b146f"
server_key_iv: "f95b5b40a4a41e391fca9b4d36526159"
server_key_version: "encryptionkey@0"
I20260808 18:33:01.172362 21193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:01.187973 21193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:01.191797 21193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:01.192031 21193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:01.199726 21193 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35157
I20260808 18:33:01.199798 21266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35157 every 8 connection(s)
I20260808 18:33:01.200783 21193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:01.203536 21267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:01.208905 21267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282: Bootstrap starting.
I20260808 18:33:01.210777 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 21193
I20260808 18:33:01.210938 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:33:01.211221 30010 external_mini_cluster.cc:1428] Setting key ca8c284860a2ebd71a44043bf9513e45
I20260808 18:33:01.211483 21267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:01.212385 21267 log.cc:824] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:01.214637 21267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282: No bootstrap required, opened a new log
I20260808 18:33:01.218989 21267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c6be3962b6450799cfe1a8eafab282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35157 } }
I20260808 18:33:01.219228 21267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:01.219312 21267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68c6be3962b6450799cfe1a8eafab282, State: Initialized, Role: FOLLOWER
I20260808 18:33:01.219784 21267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c6be3962b6450799cfe1a8eafab282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35157 } }
I20260808 18:33:01.219937 21267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:01.220034 21267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:01.220132 21267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:01.221108 21267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c6be3962b6450799cfe1a8eafab282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35157 } }
I20260808 18:33:01.221172 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:01.212414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:34327 (local address 127.0.0.1:35157)
0808 18:33:01.212889 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:01.212904 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:01.212945 (+ 41us) server_negotiation.cc:407] Connection header received
0808 18:33:01.213766 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:01.213788 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:01.214227 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:01.214606 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:01.215215 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:01.216728 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:01.217636 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:01.218352 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:01.218557 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:01.219476 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:01.219482 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:01.219506 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:01.219878 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:01.219975 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:01.220010 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:01.220148 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:01.220296 (+ 148us) server_negotiation.cc:299] Negotiation successful
0808 18:33:01.220439 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":139,"threads_started":1}
I20260808 18:33:01.221498 21267 leader_election.cc:302] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68c6be3962b6450799cfe1a8eafab282; no voters:
I20260808 18:33:01.221825 21267 leader_election.cc:288] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:01.221930 21273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:01.222216 21273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282 [term 1 LEADER]: Becoming Leader. State: Replica: 68c6be3962b6450799cfe1a8eafab282, State: Running, Role: LEADER
I20260808 18:33:01.222594 21273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c6be3962b6450799cfe1a8eafab282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35157 } }
I20260808 18:33:01.223126 21267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:01.223812 21274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68c6be3962b6450799cfe1a8eafab282. Latest consensus state: current_term: 1 leader_uuid: "68c6be3962b6450799cfe1a8eafab282" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c6be3962b6450799cfe1a8eafab282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35157 } } }
I20260808 18:33:01.223992 21274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:01.224237 21275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68c6be3962b6450799cfe1a8eafab282" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c6be3962b6450799cfe1a8eafab282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35157 } } }
I20260808 18:33:01.224349 21275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:01.224342 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:33:01.224439 21279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:01.227854 21279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:01.233846 21279 catalog_manager.cc:1347] Generated new cluster ID: 5495bcf8b3ad40288cab05d492b7185f
I20260808 18:33:01.233922 21279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:01.234269 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:01.226896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:54739 (local address 127.0.0.1:35157)
0808 18:33:01.227014 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:01.227018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:01.229940 (+ 2922us) server_negotiation.cc:407] Connection header received
0808 18:33:01.230110 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:01.230115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:01.230162 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:01.230249 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:01.230576 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:01.231159 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:01.232145 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:01.232416 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:01.232522 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:01.233656 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:01.233659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:01.233662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:01.233675 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:01.233703 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:01.233715 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:01.233797 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:01.234125 (+ 328us) server_negotiation.cc:299] Negotiation successful
0808 18:33:01.234163 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:33:01.249728 21279 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:01.250785 21279 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:01.261617 21279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68c6be3962b6450799cfe1a8eafab282: Generated new TSK 0
I20260808 18:33:01.262339 21279 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:33:01.269203 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 21193
W20260808 18:33:01.279165 21178 connection.cc:535] client connection to 127.0.0.1:35157 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35157): Connection reset by peer (error 104)
2026-08-08T18:33:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1518 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:33:01 dist-test-slave-gp6n krb5kdc[21304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:33:01 dist-test-slave-gp6n krb5kdc[21304](info): set up 2 sockets
Aug 08 18:33:01 dist-test-slave-gp6n krb5kdc[21304](info): commencing operation
krb5kdc: starting...
W20260808 18:33:02.370492 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:33:02 dist-test-slave-gp6n krb5kdc[21304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:33:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:33:02.513813 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45331
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:02.623425 21320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:02.623684 21320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:02.623724 21320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:02.627702 21320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:02.627761 21320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:02.627780 21320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:02.627797 21320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:02.627859 21320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:02.633159 21320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45331
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:02.634370 21320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:02.635454 21320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:02.643873 21326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:02.643865 21327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:02.643953 21320 server_base.cc:1034] running on GCE node
W20260808 18:33:02.644012 21329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:02.644768 21320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:02.645877 21320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:02.647068 21320 hybrid_clock.cc:648] HybridClock initialized: now 1786213982647040 us; error 40 us; skew 500 ppm
Aug 08 18:33:02 dist-test-slave-gp6n krb5kdc[21304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:02.649889 21320 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:33:02.650875 21320 webserver.cc:460] Webserver started at http://127.0.0.1:42915/ using document root <none> and password file <none>
I20260808 18:33:02.651402 21320 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:02.651461 21320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:02.651631 21320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:02.653637 21320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "924c561cd3674311ada5fd67f2d52361"
format_stamp: "Formatted at 2026-08-08 18:33:02 on dist-test-slave-gp6n"
server_key: "b591b9513f5468eaa8975c0b3b48861b"
server_key_iv: "01d52325091e4047fad3a86c86094043"
server_key_version: "encryptionkey@0"
I20260808 18:33:02.654152 21320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "924c561cd3674311ada5fd67f2d52361"
format_stamp: "Formatted at 2026-08-08 18:33:02 on dist-test-slave-gp6n"
server_key: "b591b9513f5468eaa8975c0b3b48861b"
server_key_iv: "01d52325091e4047fad3a86c86094043"
server_key_version: "encryptionkey@0"
I20260808 18:33:02.657826 21320 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 18:33:02.660135 21336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:02.661331 21320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:33:02.661437 21320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "924c561cd3674311ada5fd67f2d52361"
format_stamp: "Formatted at 2026-08-08 18:33:02 on dist-test-slave-gp6n"
server_key: "b591b9513f5468eaa8975c0b3b48861b"
server_key_iv: "01d52325091e4047fad3a86c86094043"
server_key_version: "encryptionkey@0"
I20260808 18:33:02.661520 21320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:02.665992 21320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:02.668911 21320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:02.669063 21320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:02.676381 21320 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45331
I20260808 18:33:02.676396 21397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45331 every 8 connection(s)
I20260808 18:33:02.677325 21320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:02.679441 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 21320
I20260808 18:33:02.679603 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:33:02.679896 30010 external_mini_cluster.cc:1428] Setting key 9fbb937b157e42c082bd76211162ac31
I20260808 18:33:02.680235 21398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:02.684983 21401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:02.681218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:53087 (local address 127.0.0.1:45331)
0808 18:33:02.681622 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:02.681641 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:02.681681 (+ 40us) server_negotiation.cc:407] Connection header received
0808 18:33:02.682307 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:02.682324 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:02.682670 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:02.682943 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:02.683785 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:02.683789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:02.683801 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:02.683811 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:02.683863 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:02.683881 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:02.683990 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:02.684121 (+ 131us) server_negotiation.cc:299] Negotiation successful
0808 18:33:02.684294 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":141,"threads_started":1}
I20260808 18:33:02.686977 21398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361: Bootstrap starting.
I20260808 18:33:02.687805 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:33:02.689652 21398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:02.690610 21398 log.cc:824] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:02.691665 21401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:02.690062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:57529 (local address 127.0.0.1:45331)
0808 18:33:02.690232 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:02.690235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:02.690319 (+ 84us) server_negotiation.cc:407] Connection header received
0808 18:33:02.690471 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:02.690473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:02.690522 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:02.690585 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:02.691264 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:02.691267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:02.691270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:02.691281 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:02.691305 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:02.691318 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:02.691411 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:02.691518 (+ 107us) server_negotiation.cc:299] Negotiation successful
0808 18:33:02.691556 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260808 18:33:02.692687 21398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361: No bootstrap required, opened a new log
I20260808 18:33:02.696246 21398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924c561cd3674311ada5fd67f2d52361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45331 } }
I20260808 18:33:02.696511 21398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:02.696575 21398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 924c561cd3674311ada5fd67f2d52361, State: Initialized, Role: FOLLOWER
I20260808 18:33:02.696966 21398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924c561cd3674311ada5fd67f2d52361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45331 } }
I20260808 18:33:02.697099 21398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:02.697146 21398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:02.697216 21398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:02.698002 21398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924c561cd3674311ada5fd67f2d52361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45331 } }
I20260808 18:33:02.698275 21398 leader_election.cc:302] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 924c561cd3674311ada5fd67f2d52361; no voters:
I20260808 18:33:02.698575 21398 leader_election.cc:288] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:02.698658 21408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:02.698921 21408 raft_consensus.cc:686] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361 [term 1 LEADER]: Becoming Leader. State: Replica: 924c561cd3674311ada5fd67f2d52361, State: Running, Role: LEADER
I20260808 18:33:02.699239 21408 consensus_queue.cc:237] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924c561cd3674311ada5fd67f2d52361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45331 } }
I20260808 18:33:02.699759 21398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:02.700824 21410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 924c561cd3674311ada5fd67f2d52361. Latest consensus state: current_term: 1 leader_uuid: "924c561cd3674311ada5fd67f2d52361" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924c561cd3674311ada5fd67f2d52361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45331 } } }
I20260808 18:33:02.700953 21410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:02.701321 21417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:02.701288 21409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "924c561cd3674311ada5fd67f2d52361" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924c561cd3674311ada5fd67f2d52361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45331 } } }
I20260808 18:33:02.701951 21409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:02.704471 21417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:02.709625 21417 catalog_manager.cc:1347] Generated new cluster ID: 872460090ef7426ba85c565684073278
I20260808 18:33:02.709681 21417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:02.736614 21417 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:02.736984 21417 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:02.744415 21417 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 924c561cd3674311ada5fd67f2d52361: Generated new TSK 0
I20260808 18:33:02.745163 21417 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:33:02.749862 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 21320
2026-08-08T18:33:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1477 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:33:02 dist-test-slave-gp6n krb5kdc[21433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:33:02 dist-test-slave-gp6n krb5kdc[21433](info): set up 2 sockets
Aug 08 18:33:02 dist-test-slave-gp6n krb5kdc[21433](info): commencing operation
krb5kdc: starting...
W20260808 18:33:03.858116 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:33:03 dist-test-slave-gp6n krb5kdc[21433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:33:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:33:03.987658 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:35869
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:34647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:35869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:04.100914 21450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:04.101190 21450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:04.101262 21450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:04.105415 21450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:04.105480 21450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:04.105499 21450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:04.105516 21450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:04.105577 21450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:04.110891 21450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:34647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:35869
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:35869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:04.111953 21450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:04.113031 21450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:04.121822 21458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:04.121795 21456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:04.121840 21455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:04.121920 21450 server_base.cc:1034] running on GCE node
I20260808 18:33:04.122730 21450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:04.123854 21450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:04.125056 21450 hybrid_clock.cc:648] HybridClock initialized: now 1786213984125037 us; error 33 us; skew 500 ppm
Aug 08 18:33:04 dist-test-slave-gp6n krb5kdc[21433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213984, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:04.127894 21450 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:33:04.128810 21450 webserver.cc:460] Webserver started at http://127.29.78.190:34189/ using document root <none> and password file <none>
I20260808 18:33:04.129421 21450 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:04.129485 21450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:04.129704 21450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:04.131829 21450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "c5d7464203c248969618b2b1d3d939a4"
format_stamp: "Formatted at 2026-08-08 18:33:04 on dist-test-slave-gp6n"
server_key: "0169cde0c8a52aea01dc75dc348d9e23"
server_key_iv: "ea44be46a244099f531f39cfdd9b3491"
server_key_version: "encryptionkey@0"
I20260808 18:33:04.132457 21450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "c5d7464203c248969618b2b1d3d939a4"
format_stamp: "Formatted at 2026-08-08 18:33:04 on dist-test-slave-gp6n"
server_key: "0169cde0c8a52aea01dc75dc348d9e23"
server_key_iv: "ea44be46a244099f531f39cfdd9b3491"
server_key_version: "encryptionkey@0"
I20260808 18:33:04.136260 21450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260808 18:33:04.138757 21465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:04.139912 21450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:33:04.140038 21450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "c5d7464203c248969618b2b1d3d939a4"
format_stamp: "Formatted at 2026-08-08 18:33:04 on dist-test-slave-gp6n"
server_key: "0169cde0c8a52aea01dc75dc348d9e23"
server_key_iv: "ea44be46a244099f531f39cfdd9b3491"
server_key_version: "encryptionkey@0"
I20260808 18:33:04.140139 21450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:04.162535 21450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:04.165760 21450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:04.165982 21450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:04.173565 21450 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:35869
I20260808 18:33:04.173585 21522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:35869 every 8 connection(s)
I20260808 18:33:04.174604 21450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:04.175050 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 21450
I20260808 18:33:04.175208 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:33:04.175515 30010 external_mini_cluster.cc:1428] Setting key 2b43e7cae28f00c02bf65ff61ea7b409
I20260808 18:33:04.177702 21523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:04.184211 21523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4: Bootstrap starting.
Aug 08 18:33:04 dist-test-slave-gp6n krb5kdc[21433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:33:04.186862 21523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:04.187772 21523 log.cc:824] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:04.190193 21523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4: No bootstrap required, opened a new log
I20260808 18:33:04.191028 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:04.177009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59631 (local address 127.29.78.190:35869)
0808 18:33:04.177410 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:04.177431 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:04.177473 (+ 42us) server_negotiation.cc:407] Connection header received
0808 18:33:04.178183 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:04.178201 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:04.178598 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:04.178881 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:04.179786 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:04.180860 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:04.182036 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:04.182570 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:04.182688 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:04.185587 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:04.185593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:04.185608 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:04.185639 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:04.187777 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:04.188321 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:04.188330 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:04.188338 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:04.188500 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:04.189009 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:04.189011 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:04.189012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:04.189406 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:04.189572 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:04.189930 (+ 358us) server_negotiation.cc:299] Negotiation successful
0808 18:33:04.190182 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":137,"threads_started":1}
I20260808 18:33:04.193879 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:33:04.193774 21523 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d7464203c248969618b2b1d3d939a4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35869 } }
I20260808 18:33:04.194064 21523 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:04.194118 21523 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d7464203c248969618b2b1d3d939a4, State: Initialized, Role: FOLLOWER
I20260808 18:33:04.194599 21523 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d7464203c248969618b2b1d3d939a4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35869 } }
I20260808 18:33:04.194772 21523 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:04.194859 21523 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:04.194957 21523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:04.196177 21523 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d7464203c248969618b2b1d3d939a4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35869 } }
I20260808 18:33:04.196549 21523 leader_election.cc:302] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5d7464203c248969618b2b1d3d939a4; no voters:
I20260808 18:33:04.196933 21523 leader_election.cc:288] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:04.197036 21534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:04.197247 21534 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4 [term 1 LEADER]: Becoming Leader. State: Replica: c5d7464203c248969618b2b1d3d939a4, State: Running, Role: LEADER
I20260808 18:33:04.197558 21534 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d7464203c248969618b2b1d3d939a4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35869 } }
I20260808 18:33:04.198167 21523 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:04.198732 21534 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5d7464203c248969618b2b1d3d939a4. Latest consensus state: current_term: 1 leader_uuid: "c5d7464203c248969618b2b1d3d939a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d7464203c248969618b2b1d3d939a4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35869 } } }
I20260808 18:33:04.198863 21534 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:04.199173 21535 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5d7464203c248969618b2b1d3d939a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d7464203c248969618b2b1d3d939a4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35869 } } }
I20260808 18:33:04.199270 21535 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:04.199568 21542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:04.203356 21542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:04.205708 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:04.196230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57117 (local address 127.29.78.190:35869)
0808 18:33:04.196344 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:04.196348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:04.196506 (+ 158us) server_negotiation.cc:407] Connection header received
0808 18:33:04.196602 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:04.196604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:04.196639 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:04.196735 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:04.197556 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:04.198324 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:04.199754 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:04.200158 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:04.200237 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:04.201997 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:04.202001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:04.202003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:04.202031 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:04.203811 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:04.204410 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:04.204414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:04.204415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:04.204506 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:04.204849 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:04.204852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:04.204854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:04.205026 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:04.205180 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:04.205452 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 18:33:04.205544 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 18:33:04.209699 21542 catalog_manager.cc:1347] Generated new cluster ID: 9984931427304c5992cba83c8f262e25
I20260808 18:33:04.209766 21542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:04.226956 21542 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:04.228154 21542 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:04.236222 21542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5d7464203c248969618b2b1d3d939a4: Generated new TSK 0
I20260808 18:33:04.236899 21542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:33:04.242614 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 21450
W20260808 18:33:04.252838 21430 connection.cc:535] client connection to 127.29.78.190:35869 recv error: Network error: failed to read from TLS socket (remote: 127.29.78.190:35869): Connection reset by peer (error 104)
2026-08-08T18:33:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1496 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:33:04 dist-test-slave-gp6n krb5kdc[21559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:33:04 dist-test-slave-gp6n krb5kdc[21559](info): set up 2 sockets
Aug 08 18:33:04 dist-test-slave-gp6n krb5kdc[21559](info): commencing operation
krb5kdc: starting...
W20260808 18:33:05.335598 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:33:05 dist-test-slave-gp6n krb5kdc[21559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:33:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:33:05.465770 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:37567
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:32825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:37567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:05.576715 21572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:05.577015 21572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:05.577090 21572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:05.581187 21572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:05.581264 21572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:05.581300 21572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:05.581363 21572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:05.581404 21572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:05.586553 21572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:32825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:37567
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:37567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:05.587576 21572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:05.588575 21572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:05.597230 21579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:05.597249 21572 server_base.cc:1034] running on GCE node
W20260808 18:33:05.597270 21578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:05.597266 21581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:05.597994 21572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:05.599112 21572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:05.600342 21572 hybrid_clock.cc:648] HybridClock initialized: now 1786213985600309 us; error 43 us; skew 500 ppm
Aug 08 18:33:05 dist-test-slave-gp6n krb5kdc[21559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213985, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:05.608247 21572 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:33:05.609160 21572 webserver.cc:460] Webserver started at http://127.29.78.190:33767/ using document root <none> and password file <none>
I20260808 18:33:05.609717 21572 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:05.609807 21572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:05.610047 21572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:05.612228 21572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "1e66ecea3a0848909ec155958fea23f2"
format_stamp: "Formatted at 2026-08-08 18:33:05 on dist-test-slave-gp6n"
server_key: "65c0a9da56a486cab18dda6929179598"
server_key_iv: "e45188c924e778b5fd282b97a96df65e"
server_key_version: "encryptionkey@0"
I20260808 18:33:05.612797 21572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "1e66ecea3a0848909ec155958fea23f2"
format_stamp: "Formatted at 2026-08-08 18:33:05 on dist-test-slave-gp6n"
server_key: "65c0a9da56a486cab18dda6929179598"
server_key_iv: "e45188c924e778b5fd282b97a96df65e"
server_key_version: "encryptionkey@0"
I20260808 18:33:05.616420 21572 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 18:33:05.618721 21590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:05.619818 21572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:33:05.619951 21572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "1e66ecea3a0848909ec155958fea23f2"
format_stamp: "Formatted at 2026-08-08 18:33:05 on dist-test-slave-gp6n"
server_key: "65c0a9da56a486cab18dda6929179598"
server_key_iv: "e45188c924e778b5fd282b97a96df65e"
server_key_version: "encryptionkey@0"
I20260808 18:33:05.620070 21572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:05.650103 21572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:05.653267 21572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:05.653478 21572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:05.660775 21572 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:37567
I20260808 18:33:05.660804 21645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:37567 every 8 connection(s)
I20260808 18:33:05.661703 21572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:05.662333 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 21572
I20260808 18:33:05.662489 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:33:05.662796 30010 external_mini_cluster.cc:1428] Setting key 4fea83f07c8eace09ba7f043033dbfb2
I20260808 18:33:05.664855 21646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:05.671653 21646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2: Bootstrap starting.
I20260808 18:33:05.672974 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:05.664218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56393 (local address 127.29.78.190:37567)
0808 18:33:05.664642 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:05.664666 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:05.664708 (+ 42us) server_negotiation.cc:407] Connection header received
0808 18:33:05.665552 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:05.665573 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:05.665983 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:05.666333 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:05.666774 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:05.668332 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:05.669214 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:05.669930 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:05.670084 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:05.670985 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:05.670992 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:05.671007 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:05.671495 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:05.671559 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:05.671588 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:05.671738 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:05.671944 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 18:33:05.672085 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":119,"threads_started":1}
I20260808 18:33:05.674021 21646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:05.674906 21646 log.cc:824] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:05.675997 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:33:05.677067 21646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2: No bootstrap required, opened a new log
I20260808 18:33:05.680274 21646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e66ecea3a0848909ec155958fea23f2" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37567 } }
I20260808 18:33:05.680478 21646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:05.680539 21646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e66ecea3a0848909ec155958fea23f2, State: Initialized, Role: FOLLOWER
I20260808 18:33:05.680994 21646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e66ecea3a0848909ec155958fea23f2" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37567 } }
I20260808 18:33:05.681146 21646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:05.681221 21646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:05.681322 21646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:05.682307 21646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e66ecea3a0848909ec155958fea23f2" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37567 } }
I20260808 18:33:05.682533 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:05.678265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41169 (local address 127.29.78.190:37567)
0808 18:33:05.678394 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:05.678420 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:05.678647 (+ 227us) server_negotiation.cc:407] Connection header received
0808 18:33:05.678770 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:05.678774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:05.678827 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:05.678911 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:05.679377 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:05.679846 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:05.680793 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:05.681025 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:05.681139 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:05.682007 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:05.682010 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:05.682013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:05.682033 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:05.682056 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:05.682067 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:05.682167 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:05.682291 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 18:33:05.682325 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 18:33:05.682729 21646 leader_election.cc:302] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e66ecea3a0848909ec155958fea23f2; no voters:
I20260808 18:33:05.683063 21646 leader_election.cc:288] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:05.683185 21660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:05.683398 21660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2 [term 1 LEADER]: Becoming Leader. State: Replica: 1e66ecea3a0848909ec155958fea23f2, State: Running, Role: LEADER
I20260808 18:33:05.683681 21660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e66ecea3a0848909ec155958fea23f2" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37567 } }
I20260808 18:33:05.684316 21646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:05.684741 21660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e66ecea3a0848909ec155958fea23f2. Latest consensus state: current_term: 1 leader_uuid: "1e66ecea3a0848909ec155958fea23f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e66ecea3a0848909ec155958fea23f2" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37567 } } }
I20260808 18:33:05.684855 21660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:05.685209 21662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e66ecea3a0848909ec155958fea23f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e66ecea3a0848909ec155958fea23f2" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37567 } } }
I20260808 18:33:05.685328 21662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:05.686615 21664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:05.689640 21664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:05.694861 21664 catalog_manager.cc:1347] Generated new cluster ID: 4db925bedc8647088c3aa2668ca0a4f9
I20260808 18:33:05.694942 21664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:05.704873 21664 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:05.705721 21664 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:05.709316 21664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e66ecea3a0848909ec155958fea23f2: Generated new TSK 0
I20260808 18:33:05.709916 21664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:33:05.713502 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 21572
W20260808 18:33:05.722437 21557 connection.cc:535] client connection to 127.29.78.190:37567 recv error: Network error: failed to read from TLS socket (remote: 127.29.78.190:37567): Connection reset by peer (error 104)
2026-08-08T18:33:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1469 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:33:05 dist-test-slave-gp6n krb5kdc[21684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:33:05 dist-test-slave-gp6n krb5kdc[21684](info): set up 2 sockets
Aug 08 18:33:05 dist-test-slave-gp6n krb5kdc[21684](info): commencing operation
krb5kdc: starting...
W20260808 18:33:06.833521 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:33:06 dist-test-slave-gp6n krb5kdc[21684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:33:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:33:06.962936 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:41337
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:41337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:07.074254 21699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:07.074553 21699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:07.074596 21699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:07.078727 21699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:07.078789 21699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:07.078809 21699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:07.078825 21699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:07.078871 21699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:07.084113 21699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:41337
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:41337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:07.085186 21699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:07.086247 21699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:07.095475 21708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:07.095423 21710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:07.095633 21699 server_base.cc:1034] running on GCE node
W20260808 18:33:07.095398 21707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:07.096318 21699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:07.097438 21699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:07.098631 21699 hybrid_clock.cc:648] HybridClock initialized: now 1786213987098612 us; error 32 us; skew 500 ppm
Aug 08 18:33:07 dist-test-slave-gp6n krb5kdc[21684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213987, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:07.101430 21699 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:33:07.102293 21699 webserver.cc:460] Webserver started at http://127.29.78.190:35067/ using document root <none> and password file <none>
I20260808 18:33:07.102882 21699 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:07.103000 21699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:07.103235 21699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:07.105329 21699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "dca175dd6a2047ac9fad9d55412f5f45"
format_stamp: "Formatted at 2026-08-08 18:33:07 on dist-test-slave-gp6n"
server_key: "26661469930f27ac68391990c2194d8d"
server_key_iv: "8a672f494d0571b39a35e925d94fafa9"
server_key_version: "encryptionkey@0"
I20260808 18:33:07.105938 21699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "dca175dd6a2047ac9fad9d55412f5f45"
format_stamp: "Formatted at 2026-08-08 18:33:07 on dist-test-slave-gp6n"
server_key: "26661469930f27ac68391990c2194d8d"
server_key_iv: "8a672f494d0571b39a35e925d94fafa9"
server_key_version: "encryptionkey@0"
I20260808 18:33:07.109834 21699 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 18:33:07.112406 21717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:07.113765 21699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:33:07.113958 21699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "dca175dd6a2047ac9fad9d55412f5f45"
format_stamp: "Formatted at 2026-08-08 18:33:07 on dist-test-slave-gp6n"
server_key: "26661469930f27ac68391990c2194d8d"
server_key_iv: "8a672f494d0571b39a35e925d94fafa9"
server_key_version: "encryptionkey@0"
I20260808 18:33:07.114089 21699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:07.119256 21699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:07.122268 21699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:07.122538 21699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:07.130046 21699 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:41337
I20260808 18:33:07.130080 21776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:41337 every 8 connection(s)
I20260808 18:33:07.131057 21699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:07.133910 21777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:07.139331 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 21699
I20260808 18:33:07.139459 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:33:07.139753 30010 external_mini_cluster.cc:1428] Setting key 0c4c3e43b9250d86421333bae83367a7
I20260808 18:33:07.139835 21777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45: Bootstrap starting.
I20260808 18:33:07.142335 21777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:07.143232 21777 log.cc:824] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:07.145085 21781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:07.141094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52115 (local address 127.29.78.190:41337)
0808 18:33:07.141535 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:07.141550 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:07.141585 (+ 35us) server_negotiation.cc:407] Connection header received
0808 18:33:07.142200 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:07.142216 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:07.142595 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:07.142873 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:07.143764 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:07.143770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:07.143785 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:07.143800 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:07.143868 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:07.143896 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:07.144041 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:07.144187 (+ 146us) server_negotiation.cc:299] Negotiation successful
0808 18:33:07.144356 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":136,"threads_started":1}
I20260808 18:33:07.145459 21777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45: No bootstrap required, opened a new log
I20260808 18:33:07.148057 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:33:07.150014 21777 raft_consensus.cc:348] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca175dd6a2047ac9fad9d55412f5f45" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41337 } }
I20260808 18:33:07.150290 21777 raft_consensus.cc:374] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:07.150355 21777 raft_consensus.cc:729] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca175dd6a2047ac9fad9d55412f5f45, State: Initialized, Role: FOLLOWER
I20260808 18:33:07.150887 21777 consensus_queue.cc:260] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca175dd6a2047ac9fad9d55412f5f45" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41337 } }
I20260808 18:33:07.151089 21777 raft_consensus.cc:388] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:07.151186 21777 raft_consensus.cc:482] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:07.151288 21777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:07.152419 21781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:07.150518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41809 (local address 127.29.78.190:41337)
0808 18:33:07.150664 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:07.150669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:07.150919 (+ 250us) server_negotiation.cc:407] Connection header received
0808 18:33:07.151008 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:07.151010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:07.151068 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:07.151187 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:07.151981 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:07.151985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:07.151987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:07.151999 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:07.152033 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:07.152044 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:07.152134 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:07.152216 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 18:33:07.152254 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 18:33:07.152395 21777 raft_consensus.cc:504] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca175dd6a2047ac9fad9d55412f5f45" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41337 } }
I20260808 18:33:07.152783 21777 leader_election.cc:302] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dca175dd6a2047ac9fad9d55412f5f45; no voters:
I20260808 18:33:07.153127 21777 leader_election.cc:288] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:07.153290 21789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:07.153533 21789 raft_consensus.cc:686] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45 [term 1 LEADER]: Becoming Leader. State: Replica: dca175dd6a2047ac9fad9d55412f5f45, State: Running, Role: LEADER
I20260808 18:33:07.153833 21789 consensus_queue.cc:237] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca175dd6a2047ac9fad9d55412f5f45" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41337 } }
I20260808 18:33:07.154358 21777 sys_catalog.cc:564] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:07.155557 21791 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dca175dd6a2047ac9fad9d55412f5f45. Latest consensus state: current_term: 1 leader_uuid: "dca175dd6a2047ac9fad9d55412f5f45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca175dd6a2047ac9fad9d55412f5f45" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41337 } } }
I20260808 18:33:07.155696 21791 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:07.156015 21790 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dca175dd6a2047ac9fad9d55412f5f45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca175dd6a2047ac9fad9d55412f5f45" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41337 } } }
I20260808 18:33:07.156096 21798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:07.156124 21790 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:07.159137 21798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:07.164505 21798 catalog_manager.cc:1347] Generated new cluster ID: e4cf6d44de7748b7844246349eaaf568
I20260808 18:33:07.164572 21798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:07.173491 21798 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:07.173851 21798 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:07.179060 21798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dca175dd6a2047ac9fad9d55412f5f45: Generated new TSK 0
I20260808 18:33:07.179761 21798 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:33:07.185139 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 21699
2026-08-08T18:33:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1469 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:33:07 dist-test-slave-gp6n krb5kdc[21813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:33:07 dist-test-slave-gp6n krb5kdc[21813](info): set up 2 sockets
Aug 08 18:33:07 dist-test-slave-gp6n krb5kdc[21813](info): commencing operation
krb5kdc: starting...
W20260808 18:33:08.284729 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:33:08 dist-test-slave-gp6n krb5kdc[21813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:33:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:33:08.414391 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:36719
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:36719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:08.524652 21826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:08.524955 21826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:08.525034 21826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:08.528968 21826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:08.529055 21826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:08.529094 21826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:08.529153 21826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:08.529194 21826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:08.534325 21826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:36719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:36719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:08.535391 21826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:08.536415 21826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:08.545179 21833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:08.545286 21826 server_base.cc:1034] running on GCE node
W20260808 18:33:08.545188 21831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:08.545264 21835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:08.546046 21826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:08.547297 21826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:08.548466 21826 hybrid_clock.cc:648] HybridClock initialized: now 1786213988548447 us; error 29 us; skew 500 ppm
Aug 08 18:33:08 dist-test-slave-gp6n krb5kdc[21813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213988, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:08.551313 21826 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:33:08.552227 21826 webserver.cc:460] Webserver started at http://127.29.78.190:38043/ using document root <none> and password file <none>
I20260808 18:33:08.552747 21826 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:08.552807 21826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:08.552975 21826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:08.555079 21826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "0b456c9f9cf848c89386a10c53e167d4"
format_stamp: "Formatted at 2026-08-08 18:33:08 on dist-test-slave-gp6n"
server_key: "558decbeb913ddc929d16ebc87d21e77"
server_key_iv: "79ffbb975833ceca9cd5ecc7a670db2f"
server_key_version: "encryptionkey@0"
I20260808 18:33:08.555616 21826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "0b456c9f9cf848c89386a10c53e167d4"
format_stamp: "Formatted at 2026-08-08 18:33:08 on dist-test-slave-gp6n"
server_key: "558decbeb913ddc929d16ebc87d21e77"
server_key_iv: "79ffbb975833ceca9cd5ecc7a670db2f"
server_key_version: "encryptionkey@0"
I20260808 18:33:08.559260 21826 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:33:08.561615 21842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:08.562978 21826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:33:08.563118 21826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "0b456c9f9cf848c89386a10c53e167d4"
format_stamp: "Formatted at 2026-08-08 18:33:08 on dist-test-slave-gp6n"
server_key: "558decbeb913ddc929d16ebc87d21e77"
server_key_iv: "79ffbb975833ceca9cd5ecc7a670db2f"
server_key_version: "encryptionkey@0"
I20260808 18:33:08.563232 21826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:08.586793 21826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:08.589956 21826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:08.590167 21826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:08.597574 21826 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:36719
I20260808 18:33:08.597599 21904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:36719 every 8 connection(s)
I20260808 18:33:08.598599 21826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:08.601294 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 21826
I20260808 18:33:08.601434 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:33:08.601532 21905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:08.601749 30010 external_mini_cluster.cc:1428] Setting key 7fa7c6949339f7e303fb4496adf8345d
I20260808 18:33:08.609921 21905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4: Bootstrap starting.
Aug 08 18:33:08 dist-test-slave-gp6n krb5kdc[21813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:33:08.612507 21905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:08.613595 21905 log.cc:824] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:08.615727 21905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4: No bootstrap required, opened a new log
I20260808 18:33:08.617473 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:08.603237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59129 (local address 127.29.78.190:36719)
0808 18:33:08.603690 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:08.603723 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:08.603771 (+ 48us) server_negotiation.cc:407] Connection header received
0808 18:33:08.604700 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:08.604741 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:08.605221 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:08.605645 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:08.606460 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:08.608172 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:08.609070 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:08.609857 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:08.610041 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:08.612651 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:08.612658 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:08.612678 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:08.612717 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:08.614639 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:08.615097 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:08.615103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:08.615107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:08.615215 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:08.615526 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:08.615530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:08.615532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:08.615982 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:08.616183 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:08.616476 (+ 293us) server_negotiation.cc:299] Negotiation successful
0808 18:33:08.616768 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":149,"threads_started":1}
I20260808 18:33:08.620239 21905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b456c9f9cf848c89386a10c53e167d4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36719 } }
I20260808 18:33:08.620563 21905 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:08.620589 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:33:08.620656 21905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b456c9f9cf848c89386a10c53e167d4, State: Initialized, Role: FOLLOWER
I20260808 18:33:08.621160 21905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b456c9f9cf848c89386a10c53e167d4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36719 } }
I20260808 18:33:08.621295 21905 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:08.621392 21905 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:08.621500 21905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:08.622670 21905 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b456c9f9cf848c89386a10c53e167d4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36719 } }
I20260808 18:33:08.623071 21905 leader_election.cc:302] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b456c9f9cf848c89386a10c53e167d4; no voters:
I20260808 18:33:08.623448 21905 leader_election.cc:288] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:08.623835 21917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:08.624070 21917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4 [term 1 LEADER]: Becoming Leader. State: Replica: 0b456c9f9cf848c89386a10c53e167d4, State: Running, Role: LEADER
I20260808 18:33:08.624459 21917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b456c9f9cf848c89386a10c53e167d4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36719 } }
I20260808 18:33:08.624895 21905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:08.625875 21917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b456c9f9cf848c89386a10c53e167d4. Latest consensus state: current_term: 1 leader_uuid: "0b456c9f9cf848c89386a10c53e167d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b456c9f9cf848c89386a10c53e167d4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36719 } } }
I20260808 18:33:08.626093 21917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:08.626286 21918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b456c9f9cf848c89386a10c53e167d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b456c9f9cf848c89386a10c53e167d4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36719 } } }
I20260808 18:33:08.626395 21918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:08.626703 21925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:08.630272 21925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:08.631654 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:08.622839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44089 (local address 127.29.78.190:36719)
0808 18:33:08.622962 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:08.622969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:08.623185 (+ 216us) server_negotiation.cc:407] Connection header received
0808 18:33:08.623305 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:08.623310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:08.623359 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:08.623443 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:08.624201 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:08.624885 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:08.625910 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:08.626346 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:08.626451 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:08.628258 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:08.628261 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:08.628264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:08.628296 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:08.629795 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:08.630219 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:08.630223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:08.630225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:08.630308 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:08.630740 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:08.630744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:08.630745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:08.630982 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:08.631129 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:08.631372 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 18:33:08.631486 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 18:33:08.637985 21925 catalog_manager.cc:1347] Generated new cluster ID: af5a66a55a3046ec8317f433e035cb7d
I20260808 18:33:08.638049 21925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:08.668864 21925 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:08.670208 21925 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:08.681217 21925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b456c9f9cf848c89386a10c53e167d4: Generated new TSK 0
I20260808 18:33:08.682049 21925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:33:08.696763 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 21826
2026-08-08T18:33:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1512 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:33:08 dist-test-slave-gp6n krb5kdc[21939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:33:08 dist-test-slave-gp6n krb5kdc[21939](info): set up 2 sockets
Aug 08 18:33:08 dist-test-slave-gp6n krb5kdc[21939](info): commencing operation
krb5kdc: starting...
W20260808 18:33:09.845096 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:33:09 dist-test-slave-gp6n krb5kdc[21939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:33:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:33:09.983109 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:45633
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:45633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:10.102531 21954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:10.102851 21954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:10.102928 21954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:10.107188 21954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:10.107313 21954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:10.107352 21954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:10.107393 21954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:10.107425 21954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:10.112972 21954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:45633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:45633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:10.114115 21954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:10.115231 21954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:10.124106 21963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:10.124099 21962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:10.124239 21954 server_base.cc:1034] running on GCE node
W20260808 18:33:10.124099 21966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:10.124998 21954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:10.126129 21954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:10.127319 21954 hybrid_clock.cc:648] HybridClock initialized: now 1786213990127304 us; error 45 us; skew 500 ppm
Aug 08 18:33:10 dist-test-slave-gp6n krb5kdc[21939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213990, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:10.130254 21954 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:33:10.131193 21954 webserver.cc:460] Webserver started at http://127.29.78.190:36807/ using document root <none> and password file <none>
I20260808 18:33:10.131757 21954 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:10.131850 21954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:10.132090 21954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:10.134308 21954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "86b63a3813804647b48bc92b304c5415"
format_stamp: "Formatted at 2026-08-08 18:33:10 on dist-test-slave-gp6n"
server_key: "8a1a1a568f74642fe4661bcde0fd05eb"
server_key_iv: "80f6e0cf55fa08cfde8f3729eb79eacf"
server_key_version: "encryptionkey@0"
I20260808 18:33:10.134946 21954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "86b63a3813804647b48bc92b304c5415"
format_stamp: "Formatted at 2026-08-08 18:33:10 on dist-test-slave-gp6n"
server_key: "8a1a1a568f74642fe4661bcde0fd05eb"
server_key_iv: "80f6e0cf55fa08cfde8f3729eb79eacf"
server_key_version: "encryptionkey@0"
I20260808 18:33:10.138739 21954 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:33:10.141203 21973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:10.142575 21954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:33:10.142714 21954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "86b63a3813804647b48bc92b304c5415"
format_stamp: "Formatted at 2026-08-08 18:33:10 on dist-test-slave-gp6n"
server_key: "8a1a1a568f74642fe4661bcde0fd05eb"
server_key_iv: "80f6e0cf55fa08cfde8f3729eb79eacf"
server_key_version: "encryptionkey@0"
I20260808 18:33:10.142832 21954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:10.161137 21954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:10.164215 21954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:10.164444 21954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:10.172093 21954 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:45633
I20260808 18:33:10.172115 22034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:45633 every 8 connection(s)
I20260808 18:33:10.173094 21954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:10.175859 22035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:10.180858 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 21954
I20260808 18:33:10.181002 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:33:10.181314 30010 external_mini_cluster.cc:1428] Setting key a030307ca55e4e05ce4c31e7cad72fc1
I20260808 18:33:10.181851 22035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415: Bootstrap starting.
I20260808 18:33:10.184350 22035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:10.185106 22035 log.cc:824] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:10.187063 22035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415: No bootstrap required, opened a new log
I20260808 18:33:10.190352 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:10.182787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50857 (local address 127.29.78.190:45633)
0808 18:33:10.183247 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:10.183262 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:10.183298 (+ 36us) server_negotiation.cc:407] Connection header received
0808 18:33:10.183938 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:10.183954 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:10.184306 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:10.184670 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:10.185098 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:10.186108 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:10.187090 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:10.187614 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:10.187791 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:10.188672 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:10.188677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:10.188690 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:10.189068 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:10.189124 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:10.189147 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:10.189257 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:10.189426 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 18:33:10.189634 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":144,"threads_started":1}
I20260808 18:33:10.191020 22035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b63a3813804647b48bc92b304c5415" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45633 } }
I20260808 18:33:10.191285 22035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:10.191334 22035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b63a3813804647b48bc92b304c5415, State: Initialized, Role: FOLLOWER
I20260808 18:33:10.191797 22035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b63a3813804647b48bc92b304c5415" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45633 } }
I20260808 18:33:10.191929 22035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:10.191977 22035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:10.192049 22035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:10.193125 22035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b63a3813804647b48bc92b304c5415" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45633 } }
I20260808 18:33:10.193506 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:33:10.193486 22035 leader_election.cc:302] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86b63a3813804647b48bc92b304c5415; no voters:
I20260808 18:33:10.193861 22035 leader_election.cc:288] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:10.193899 22040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:10.194097 22040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415 [term 1 LEADER]: Becoming Leader. State: Replica: 86b63a3813804647b48bc92b304c5415, State: Running, Role: LEADER
I20260808 18:33:10.194500 22040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b63a3813804647b48bc92b304c5415" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45633 } }
I20260808 18:33:10.195160 22035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:10.195735 22042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86b63a3813804647b48bc92b304c5415" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b63a3813804647b48bc92b304c5415" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45633 } } }
I20260808 18:33:10.195936 22042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:10.196209 22041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86b63a3813804647b48bc92b304c5415. Latest consensus state: current_term: 1 leader_uuid: "86b63a3813804647b48bc92b304c5415" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b63a3813804647b48bc92b304c5415" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45633 } } }
I20260808 18:33:10.196326 22041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:10.196427 22054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:10.199720 22054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:10.201171 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:10.195757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60877 (local address 127.29.78.190:45633)
0808 18:33:10.195870 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:10.195876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:10.196884 (+ 1008us) server_negotiation.cc:407] Connection header received
0808 18:33:10.197066 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:10.197069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:10.197119 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:10.197169 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:10.197620 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:10.198345 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:10.199547 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:10.199814 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:10.199915 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:10.200750 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:10.200756 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:10.200759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:10.200774 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:10.200805 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:10.200821 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:10.200886 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:10.200982 (+ 96us) server_negotiation.cc:299] Negotiation successful
0808 18:33:10.201032 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 18:33:10.205673 22054 catalog_manager.cc:1347] Generated new cluster ID: 7576e72c04a341a8bf601a0ec9173b03
I20260808 18:33:10.205775 22054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:10.210368 22054 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:10.211289 22054 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:10.216491 22054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86b63a3813804647b48bc92b304c5415: Generated new TSK 0
I20260808 18:33:10.217200 22054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:33:10.223416 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 21954
2026-08-08T18:33:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1528 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:33:10 dist-test-slave-gp6n krb5kdc[22069](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:33:10 dist-test-slave-gp6n krb5kdc[22069](info): set up 2 sockets
Aug 08 18:33:10 dist-test-slave-gp6n krb5kdc[22069](info): commencing operation
krb5kdc: starting...
W20260808 18:33:11.329794 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:33:11 dist-test-slave-gp6n krb5kdc[22069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:33:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:33:11.457458 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:46369
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:33947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:46369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:11.567988 22082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:11.568222 22082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:11.568264 22082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:11.572109 22082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:11.572157 22082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:11.572175 22082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:11.572191 22082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:11.572206 22082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:11.577096 22082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:33947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:46369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:46369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:11.577968 22082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:11.578902 22082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:11.586951 22092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:11.587018 22090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:11.587067 22082 server_base.cc:1034] running on GCE node
W20260808 18:33:11.586951 22089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:11.587757 22082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:11.588807 22082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:11.590013 22082 hybrid_clock.cc:648] HybridClock initialized: now 1786213991589982 us; error 43 us; skew 500 ppm
Aug 08 18:33:11 dist-test-slave-gp6n krb5kdc[22069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213991, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:11.597939 22082 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:33:11.598865 22082 webserver.cc:460] Webserver started at http://127.29.78.190:41377/ using document root <none> and password file <none>
I20260808 18:33:11.599368 22082 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:11.599426 22082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:11.599583 22082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:11.601540 22082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "fe75c52e3462400e97f1ea18f293cbc9"
format_stamp: "Formatted at 2026-08-08 18:33:11 on dist-test-slave-gp6n"
server_key: "75ee76469018b8a691efbe23e5be15ae"
server_key_iv: "c7a0a25debba6c0853e3fe9bcba392c0"
server_key_version: "encryptionkey@0"
I20260808 18:33:11.602061 22082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "fe75c52e3462400e97f1ea18f293cbc9"
format_stamp: "Formatted at 2026-08-08 18:33:11 on dist-test-slave-gp6n"
server_key: "75ee76469018b8a691efbe23e5be15ae"
server_key_iv: "c7a0a25debba6c0853e3fe9bcba392c0"
server_key_version: "encryptionkey@0"
I20260808 18:33:11.606129 22082 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 18:33:11.608496 22099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:11.609701 22082 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:33:11.609829 22082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "fe75c52e3462400e97f1ea18f293cbc9"
format_stamp: "Formatted at 2026-08-08 18:33:11 on dist-test-slave-gp6n"
server_key: "75ee76469018b8a691efbe23e5be15ae"
server_key_iv: "c7a0a25debba6c0853e3fe9bcba392c0"
server_key_version: "encryptionkey@0"
I20260808 18:33:11.609933 22082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:11.614539 22082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:11.617640 22082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:11.617847 22082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:11.625025 22082 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:46369
I20260808 18:33:11.625054 22158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:46369 every 8 connection(s)
I20260808 18:33:11.625953 22082 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:11.628604 22159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:11.633623 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 22082
I20260808 18:33:11.633759 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:33:11.634043 30010 external_mini_cluster.cc:1428] Setting key 5fc45c6cba32928cbbc59409cf943f84
I20260808 18:33:11.634249 22159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9: Bootstrap starting.
I20260808 18:33:11.636888 22159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:11.637730 22159 log.cc:824] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:11.639671 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:11.635385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51861 (local address 127.29.78.190:46369)
0808 18:33:11.635811 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:11.635829 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:11.635871 (+ 42us) server_negotiation.cc:407] Connection header received
0808 18:33:11.636657 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:11.636680 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:11.637036 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:11.637407 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:11.638120 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:11.638125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:11.638140 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:11.638152 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:11.638214 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:11.638234 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:11.638317 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:11.638605 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 18:33:11.638751 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":104,"threads_started":1}
I20260808 18:33:11.639998 22159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9: No bootstrap required, opened a new log
I20260808 18:33:11.642315 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:33:11.643952 22159 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe75c52e3462400e97f1ea18f293cbc9" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46369 } }
I20260808 18:33:11.644232 22159 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:11.644294 22159 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe75c52e3462400e97f1ea18f293cbc9, State: Initialized, Role: FOLLOWER
I20260808 18:33:11.644755 22159 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe75c52e3462400e97f1ea18f293cbc9" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46369 } }
I20260808 18:33:11.644929 22159 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:11.645010 22159 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:11.645117 22159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:11.646112 22159 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe75c52e3462400e97f1ea18f293cbc9" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46369 } }
I20260808 18:33:11.646533 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:11.644730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52067 (local address 127.29.78.190:46369)
0808 18:33:11.644856 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:11.644861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:11.645161 (+ 300us) server_negotiation.cc:407] Connection header received
0808 18:33:11.645202 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:11.645205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:11.645240 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:11.645317 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:11.646045 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:11.646052 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:11.646058 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:11.646073 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:11.646103 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:11.646116 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:11.646218 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:11.646334 (+ 116us) server_negotiation.cc:299] Negotiation successful
0808 18:33:11.646420 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 18:33:11.646507 22159 leader_election.cc:302] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe75c52e3462400e97f1ea18f293cbc9; no voters:
I20260808 18:33:11.646836 22159 leader_election.cc:288] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:11.646943 22170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:11.647156 22170 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9 [term 1 LEADER]: Becoming Leader. State: Replica: fe75c52e3462400e97f1ea18f293cbc9, State: Running, Role: LEADER
I20260808 18:33:11.647540 22170 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe75c52e3462400e97f1ea18f293cbc9" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46369 } }
I20260808 18:33:11.648039 22159 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:11.648653 22170 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe75c52e3462400e97f1ea18f293cbc9. Latest consensus state: current_term: 1 leader_uuid: "fe75c52e3462400e97f1ea18f293cbc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe75c52e3462400e97f1ea18f293cbc9" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46369 } } }
I20260808 18:33:11.648779 22170 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:11.649080 22171 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe75c52e3462400e97f1ea18f293cbc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe75c52e3462400e97f1ea18f293cbc9" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46369 } } }
I20260808 18:33:11.649174 22171 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:11.650573 22177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:11.652998 22177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:11.658178 22177 catalog_manager.cc:1347] Generated new cluster ID: 9d4c88a14a2b4890956484960369fca2
I20260808 18:33:11.658233 22177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:11.681912 22177 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:11.682282 22177 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:11.688135 22177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe75c52e3462400e97f1ea18f293cbc9: Generated new TSK 0
I20260808 18:33:11.688795 22177 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:33:11.694555 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 22082
2026-08-08T18:33:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1470 ms)
[----------] 18 tests from AuthTokenIssuingTest (27162 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:33:11 dist-test-slave-gp6n krb5kdc[22194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:33:11 dist-test-slave-gp6n krb5kdc[22194](info): set up 2 sockets
Aug 08 18:33:11 dist-test-slave-gp6n krb5kdc[22194](info): commencing operation
krb5kdc: starting...
W20260808 18:33:12.835430 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:33:12 dist-test-slave-gp6n krb5kdc[22194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:33:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:33:12.966398 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:45565
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:45565,127.29.78.189:43685,127.29.78.188:33261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:13.077168 22209 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 18:33:13.077512 22209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:13.077589 22209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:13.077670 22209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:13.081606 22209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:13.081694 22209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:13.081734 22209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:13.081771 22209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:13.081806 22209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:13.086876 22209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:45565,127.29.78.189:43685,127.29.78.188:33261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:45565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:13.087903 22209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:13.088925 22209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:13.097764 22215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:13.097766 22214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:13.097848 22217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:13.098215 22209 server_base.cc:1034] running on GCE node
I20260808 18:33:13.098930 22209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:13.099956 22209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:13.101121 22209 hybrid_clock.cc:648] HybridClock initialized: now 1786213993101103 us; error 31 us; skew 500 ppm
Aug 08 18:33:13 dist-test-slave-gp6n krb5kdc[22194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213993, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:13.103920 22209 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:33:13.104868 22209 webserver.cc:460] Webserver started at http://127.29.78.190:38051/ using document root <none> and password file <none>
I20260808 18:33:13.105414 22209 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:13.105510 22209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:13.105749 22209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:13.107841 22209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "401f60a39d854f4c82072cf622d8a48e"
format_stamp: "Formatted at 2026-08-08 18:33:13 on dist-test-slave-gp6n"
server_key: "0bd820cd68f8ca3ba4fa2ee4e0fe8ec2"
server_key_iv: "0e9301205e54a853928fab6e8916ef34"
server_key_version: "encryptionkey@0"
I20260808 18:33:13.108409 22209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "401f60a39d854f4c82072cf622d8a48e"
format_stamp: "Formatted at 2026-08-08 18:33:13 on dist-test-slave-gp6n"
server_key: "0bd820cd68f8ca3ba4fa2ee4e0fe8ec2"
server_key_iv: "0e9301205e54a853928fab6e8916ef34"
server_key_version: "encryptionkey@0"
I20260808 18:33:13.112277 22209 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:33:13.114989 22225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:13.116314 22209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:33:13.116490 22209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "401f60a39d854f4c82072cf622d8a48e"
format_stamp: "Formatted at 2026-08-08 18:33:13 on dist-test-slave-gp6n"
server_key: "0bd820cd68f8ca3ba4fa2ee4e0fe8ec2"
server_key_iv: "0e9301205e54a853928fab6e8916ef34"
server_key_version: "encryptionkey@0"
I20260808 18:33:13.116621 22209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:13.136023 22209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:13.139114 22209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:13.139314 22209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:13.146788 22209 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:45565
I20260808 18:33:13.147087 22284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:45565 every 8 connection(s)
I20260808 18:33:13.147855 22209 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:13.150622 22285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:13.152853 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 22209
I20260808 18:33:13.152952 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:33:13.153268 30010 external_mini_cluster.cc:1428] Setting key 21f20ae742d2e0118ed004cecad4a4e8
I20260808 18:33:13.154587 22285 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45565 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.78.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.189@KRBTEST.COM" created.
Aug 08 18:33:13 dist-test-slave-gp6n krb5kdc[22194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213993, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:33:13.171311 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.156949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45565 (local address 127.0.0.1:59398)
0808 18:33:13.157495 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.157541 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:13.158523 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:13.159192 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:13.159197 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:13.159652 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:13.159883 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:13.159896 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.161032 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.161035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:13.161764 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:13.161773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.162488 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.162491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:13.162583 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.163186 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:13.163208 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:13.166033 (+ 2825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:13.168094 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:13.168108 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:13.168120 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:13.168431 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:13.168858 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:13.168861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:13.168862 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:13.168997 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:13.169881 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:13.169887 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:13.170036 (+ 149us) client_negotiation.cc:769] Sending connection context
0808 18:33:13.170184 (+ 148us) client_negotiation.cc:240] Negotiation successful
0808 18:33:13.170527 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":132,"threads_started":1}
I20260808 18:33:13.171316 22289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.157866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59398 (local address 127.29.78.190:45565)
0808 18:33:13.158181 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.158192 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.158241 (+ 49us) server_negotiation.cc:407] Connection header received
0808 18:33:13.158817 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.158827 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.159006 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.159192 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.160028 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.160905 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.161893 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.162312 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.162503 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.166211 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:13.166216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:13.166226 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:13.166248 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:13.167975 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.168563 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.168572 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.168577 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.168725 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.169102 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.169108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.169113 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.169714 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:13.169867 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:13.170312 (+ 445us) server_negotiation.cc:299] Negotiation successful
0808 18:33:13.170662 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":219,"thread_start_us":114,"threads_started":1}
I20260808 18:33:13.174037 22285 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:33:13.174862 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.174558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:43685 (local address 127.0.0.1:55658)
0808 18:33:13.174704 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.174802 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:43685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 18:33:13.175091 22226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.78.189:43685: connect: Connection refused (error 111)
I20260808 18:33:13.175920 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.175727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:43685 (local address 127.0.0.1:55672)
0808 18:33:13.175826 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.175869 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:43685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 18:33:13.176095 22285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.189:43685: Network error: Client connection negotiation failed: client connection to 127.29.78.189:43685: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.189.keytab.
Entry for principal kudu/127.29.78.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.189.keytab.
I20260808 18:33:13.183149 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.189:43685
--webserver_interface=127.29.78.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.189.keytab
--principal=kudu/127.29.78.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:45565,127.29.78.189:43685,127.29.78.188:33261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:33:13.225342 22285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } attempt: 1
I20260808 18:33:13.226512 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.226124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:43685 (local address 127.0.0.1:55676)
0808 18:33:13.226294 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.226363 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:43685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 18:33:13.227252 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.227033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:43685 (local address 127.0.0.1:55684)
0808 18:33:13.227143 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.227196 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:43685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 18:33:13.227384 22285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.189:43685: Network error: Client connection negotiation failed: client connection to 127.29.78.189:43685: connect: Connection refused (error 111)
I20260808 18:33:13.295544 22285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } attempt: 2
I20260808 18:33:13.296379 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.296061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:43685 (local address 127.0.0.1:55700)
0808 18:33:13.296213 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.296284 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:43685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 18:33:13.297235 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.296938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:43685 (local address 127.0.0.1:55704)
0808 18:33:13.297101 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.297154 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:43685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260808 18:33:13.297369 22285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.189:43685: Network error: Client connection negotiation failed: client connection to 127.29.78.189:43685: connect: Connection refused (error 111)
W20260808 18:33:13.301769 22292 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 18:33:13.302076 22292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:13.302138 22292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:13.302217 22292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:13.306257 22292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:13.306332 22292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:13.306380 22292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:13.306435 22292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:13.306478 22292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:13.311777 22292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:45565,127.29.78.189:43685,127.29.78.188:33261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.189.keytab
--principal=kudu/127.29.78.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.189:43685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.78.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:13.312819 22292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:13.313843 22292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:13.322434 22297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:13.322427 22298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:13.322572 22292 server_base.cc:1034] running on GCE node
W20260808 18:33:13.322763 22300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:13.323319 22292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:13.324369 22292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:13.325531 22292 hybrid_clock.cc:648] HybridClock initialized: now 1786213993325499 us; error 48 us; skew 500 ppm
Aug 08 18:33:13 dist-test-slave-gp6n krb5kdc[22194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213993, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:13.328393 22292 init.cc:377] Logged in from keytab as kudu/127.29.78.189@KRBTEST.COM (short username kudu)
I20260808 18:33:13.329180 22292 webserver.cc:460] Webserver started at http://127.29.78.189:42393/ using document root <none> and password file <none>
I20260808 18:33:13.329718 22292 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:13.329778 22292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:13.329939 22292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:13.331957 22292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-1/data/instance:
uuid: "4d634286b94b4cc3a69afcdad3c4aa1d"
format_stamp: "Formatted at 2026-08-08 18:33:13 on dist-test-slave-gp6n"
server_key: "38d85185ba0d6b11a99854b4a9931b65"
server_key_iv: "73605677a468044449592cf3d674171e"
server_key_version: "encryptionkey@0"
I20260808 18:33:13.332484 22292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-1/wal/instance:
uuid: "4d634286b94b4cc3a69afcdad3c4aa1d"
format_stamp: "Formatted at 2026-08-08 18:33:13 on dist-test-slave-gp6n"
server_key: "38d85185ba0d6b11a99854b4a9931b65"
server_key_iv: "73605677a468044449592cf3d674171e"
server_key_version: "encryptionkey@0"
I20260808 18:33:13.336018 22292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260808 18:33:13.338229 22309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:13.339562 22292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:33:13.339689 22292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-1/wal
uuid: "4d634286b94b4cc3a69afcdad3c4aa1d"
format_stamp: "Formatted at 2026-08-08 18:33:13 on dist-test-slave-gp6n"
server_key: "38d85185ba0d6b11a99854b4a9931b65"
server_key_iv: "73605677a468044449592cf3d674171e"
server_key_version: "encryptionkey@0"
I20260808 18:33:13.339784 22292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:13.360056 22292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:13.363253 22292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:13.363456 22292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:13.370358 22292 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.189:43685
I20260808 18:33:13.370684 22368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.189:43685 every 8 connection(s)
I20260808 18:33:13.371335 22292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-1/data/info.pb
I20260808 18:33:13.374009 22369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:13.377686 22369 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45565 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:33:13.380786 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 22292
I20260808 18:33:13.380985 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-1/wal/instance
I20260808 18:33:13.381291 30010 external_mini_cluster.cc:1428] Setting key 12f27baf9027413b83b27e9e83b9314f
Aug 08 18:33:13 dist-test-slave-gp6n krb5kdc[22194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213993, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:33:13.388556 22285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } attempt: 3
WARNING: no policy specified for kudu/127.29.78.188@KRBTEST.COM; defaulting to no policy
I20260808 18:33:13.391462 22289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.379545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59406 (local address 127.29.78.190:45565)
0808 18:33:13.379677 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.379681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.380423 (+ 742us) server_negotiation.cc:407] Connection header received
0808 18:33:13.381419 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.381423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.381488 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.381587 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.382779 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.383308 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.384443 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.384681 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.384795 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.387259 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:13.387262 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:13.387264 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:13.387287 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:13.388848 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.389554 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.389557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.389559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.389650 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.390180 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.390183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.390185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.390474 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:13.390560 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:13.391173 (+ 613us) server_negotiation.cc:299] Negotiation successful
0808 18:33:13.391288 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":47}
I20260808 18:33:13.392274 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.379811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45565 (local address 127.0.0.1:59406)
0808 18:33:13.380253 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.380295 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:13.381212 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:13.381752 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:13.381759 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:13.382104 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:13.382636 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:13.382651 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.383439 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.383450 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:13.384317 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:13.384326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.384850 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.384855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:13.385024 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.385606 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:13.385621 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:13.387090 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:13.388989 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:13.388997 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:13.389014 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:13.389429 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:13.389764 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:13.389768 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:13.389773 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:13.389995 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:13.390597 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:13.390605 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:13.390905 (+ 300us) client_negotiation.cc:769] Sending connection context
0808 18:33:13.391107 (+ 202us) client_negotiation.cc:240] Negotiation successful
0808 18:33:13.391421 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":7,"thread_start_us":113,"threads_started":1}
I20260808 18:33:13.394093 22369 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.188@KRBTEST.COM" created.
Aug 08 18:33:13 dist-test-slave-gp6n krb5kdc[22194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213993, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
Aug 08 18:33:13 dist-test-slave-gp6n krb5kdc[22194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213993, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
I20260808 18:33:13.399950 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.389053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:43685 (local address 127.0.0.1:55706)
0808 18:33:13.389230 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.389245 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:13.389342 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:13.389966 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:13.389969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:13.390170 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:13.390336 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:13.390350 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.392034 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.392038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:13.392854 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:13.392865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.393790 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.393794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:13.393890 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.394378 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:13.394392 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:13.395910 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:13.398251 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:13.398254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:13.398256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:13.398525 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:13.398830 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:13.398833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:13.398835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:13.398920 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:13.399440 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:13.399444 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:13.399553 (+ 109us) client_negotiation.cc:769] Sending connection context
0808 18:33:13.399638 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 18:33:13.399763 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":22}
I20260808 18:33:13.401234 22373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.389150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55706 (local address 127.29.78.189:43685)
0808 18:33:13.389569 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.389581 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.389604 (+ 23us) server_negotiation.cc:407] Connection header received
0808 18:33:13.389646 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.389651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.389850 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.389987 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.390504 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.391853 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.392982 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.393627 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.393767 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.396037 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:13.396046 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:13.396059 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:13.396081 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:13.398116 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.398635 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.398641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.398642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.398730 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.399016 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.399019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.399021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.399315 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:13.400236 (+ 921us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:13.400342 (+ 106us) server_negotiation.cc:299] Negotiation successful
0808 18:33:13.400489 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":324,"thread_start_us":155,"threads_started":1}
I20260808 18:33:13.403067 22285 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:33:13.403842 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.403572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33261 (local address 127.0.0.1:40668)
0808 18:33:13.403703 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.403761 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 18:33:13.404551 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.394706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:43685 (local address 127.0.0.1:55710)
0808 18:33:13.394878 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.394894 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:13.394971 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:13.395405 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:13.395408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:13.395568 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:13.395698 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:13.395705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.396841 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.396845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:13.397387 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:13.397394 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.397881 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.397885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:13.398018 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.398528 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:13.398550 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:13.400088 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:13.402782 (+ 2694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:13.402787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:13.402789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:13.403040 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:13.403309 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:13.403311 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:13.403313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:13.403403 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:13.403848 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:13.403855 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:13.404029 (+ 174us) client_negotiation.cc:769] Sending connection context
0808 18:33:13.404155 (+ 126us) client_negotiation.cc:240] Negotiation successful
0808 18:33:13.404328 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":21}
I20260808 18:33:13.404635 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.404324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33261 (local address 127.0.0.1:40682)
0808 18:33:13.404521 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.404568 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260808 18:33:13.404753 22285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:33261: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33261: connect: Connection refused (error 111)
I20260808 18:33:13.404587 22374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.394797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55710 (local address 127.29.78.189:43685)
0808 18:33:13.395111 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.395122 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.395141 (+ 19us) server_negotiation.cc:407] Connection header received
0808 18:33:13.395184 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.395190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.395245 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.395369 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.395820 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.396707 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.397510 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.397763 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.397883 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.400250 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:13.400255 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:13.400258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:13.400279 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:13.402672 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.403140 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.403142 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.403144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.403221 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.403512 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.403514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.403516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.403727 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:13.403826 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:13.404233 (+ 407us) server_negotiation.cc:299] Negotiation successful
0808 18:33:13.404399 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":229,"thread_start_us":102,"threads_started":1}
I20260808 18:33:13.405633 22369 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:33:13.406287 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.406074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33261 (local address 127.0.0.1:40696)
0808 18:33:13.406179 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.406240 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 18:33:13.406541 22310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33261: connect: Connection refused (error 111)
I20260808 18:33:13.407537 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.407240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33261 (local address 127.0.0.1:40704)
0808 18:33:13.407388 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.407434 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 18:33:13.407707 22369 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:33261: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33261: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.188.keytab.
Entry for principal kudu/127.29.78.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.188.keytab.
I20260808 18:33:13.415227 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.188:33261
--webserver_interface=127.29.78.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.188.keytab
--principal=kudu/127.29.78.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:45565,127.29.78.189:43685,127.29.78.188:33261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:33:13.435909 22285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } attempt: 1
I20260808 18:33:13.436755 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.436451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33261 (local address 127.0.0.1:40710)
0808 18:33:13.436603 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.436678 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 18:33:13.437502 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.437275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33261 (local address 127.0.0.1:40714)
0808 18:33:13.437392 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.437450 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 18:33:13.437707 22285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:33261: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33261: connect: Connection refused (error 111)
I20260808 18:33:13.456988 22369 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } attempt: 1
I20260808 18:33:13.457917 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.457581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33261 (local address 127.0.0.1:40718)
0808 18:33:13.457769 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.457836 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260808 18:33:13.458767 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.458529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33261 (local address 127.0.0.1:40722)
0808 18:33:13.458647 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.458702 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 18:33:13.458912 22369 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:33261: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33261: connect: Connection refused (error 111)
I20260808 18:33:13.512895 22285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } attempt: 2
I20260808 18:33:13.513854 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.513545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33261 (local address 127.0.0.1:40724)
0808 18:33:13.513698 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.513766 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 18:33:13.514637 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.514386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33261 (local address 127.0.0.1:40734)
0808 18:33:13.514526 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.514578 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260808 18:33:13.514772 22285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:33261: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33261: connect: Connection refused (error 111)
I20260808 18:33:13.527068 22369 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } attempt: 2
I20260808 18:33:13.527758 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.527501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33261 (local address 127.0.0.1:40744)
0808 18:33:13.527629 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.527691 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 18:33:13.527702 22377 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 18:33:13.528049 22377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:13.528111 22377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:13.528188 22377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 18:33:13.528489 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.528287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33261 (local address 127.0.0.1:40748)
0808 18:33:13.528374 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.528424 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260808 18:33:13.528616 22369 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:33261: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33261: connect: Connection refused (error 111)
W20260808 18:33:13.532361 22377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:13.532439 22377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:13.532480 22377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:13.532542 22377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:13.532577 22377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:13.537751 22377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:45565,127.29.78.189:43685,127.29.78.188:33261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.188.keytab
--principal=kudu/127.29.78.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.188:33261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.78.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:13.538848 22377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:13.539844 22377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:13.548418 22386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:13.548418 22385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:13.548411 22389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:13.548662 22377 server_base.cc:1034] running on GCE node
I20260808 18:33:13.549295 22377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:13.550386 22377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:13.551608 22377 hybrid_clock.cc:648] HybridClock initialized: now 1786213993551580 us; error 44 us; skew 500 ppm
Aug 08 18:33:13 dist-test-slave-gp6n krb5kdc[22194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213993, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:13.554373 22377 init.cc:377] Logged in from keytab as kudu/127.29.78.188@KRBTEST.COM (short username kudu)
I20260808 18:33:13.555348 22377 webserver.cc:460] Webserver started at http://127.29.78.188:44641/ using document root <none> and password file <none>
I20260808 18:33:13.555914 22377 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:13.556006 22377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:13.556234 22377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:13.558353 22377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-2/data/instance:
uuid: "412f573344b84480b888c1acf710fe46"
format_stamp: "Formatted at 2026-08-08 18:33:13 on dist-test-slave-gp6n"
server_key: "6e43e504bc280afe814d24f92fc5b5f6"
server_key_iv: "10be5284decae50ced3a2109d745fe3c"
server_key_version: "encryptionkey@0"
I20260808 18:33:13.558980 22377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-2/wal/instance:
uuid: "412f573344b84480b888c1acf710fe46"
format_stamp: "Formatted at 2026-08-08 18:33:13 on dist-test-slave-gp6n"
server_key: "6e43e504bc280afe814d24f92fc5b5f6"
server_key_iv: "10be5284decae50ced3a2109d745fe3c"
server_key_version: "encryptionkey@0"
I20260808 18:33:13.562637 22377 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:33:13.565030 22397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:13.566576 22377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:33:13.566720 22377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-2/wal
uuid: "412f573344b84480b888c1acf710fe46"
format_stamp: "Formatted at 2026-08-08 18:33:13 on dist-test-slave-gp6n"
server_key: "6e43e504bc280afe814d24f92fc5b5f6"
server_key_iv: "10be5284decae50ced3a2109d745fe3c"
server_key_version: "encryptionkey@0"
I20260808 18:33:13.566833 22377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:13.584779 22377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:13.588025 22377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:13.588233 22377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:13.595104 22377 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.188:33261
I20260808 18:33:13.595376 22453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.188:33261 every 8 connection(s)
I20260808 18:33:13.596102 22377 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-2/data/info.pb
I20260808 18:33:13.598758 22454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:13.601935 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 22377
I20260808 18:33:13.602085 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786213708769829-30010-0/minicluster-data/master-2/wal/instance
I20260808 18:33:13.602365 30010 external_mini_cluster.cc:1428] Setting key 4469cf2e960220d4ab670ed305ef9fdc
I20260808 18:33:13.602499 22454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45565 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 18:33:13 dist-test-slave-gp6n krb5kdc[22194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213993, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
Aug 08 18:33:13 dist-test-slave-gp6n krb5kdc[22194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
I20260808 18:33:13.613955 22285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } attempt: 3
I20260808 18:33:13.616469 22289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.604367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59420 (local address 127.29.78.190:45565)
0808 18:33:13.604509 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.604513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.604801 (+ 288us) server_negotiation.cc:407] Connection header received
0808 18:33:13.605258 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.605262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.605307 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.605410 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.606532 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.607325 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.608660 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.609012 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.609117 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.612006 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:13.612008 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:13.612010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:13.612035 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:13.613857 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.614597 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.614602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.614603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.614705 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.615129 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.615132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.615134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.615393 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:13.615494 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:13.616139 (+ 645us) server_negotiation.cc:299] Negotiation successful
0808 18:33:13.616256 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 18:33:13.617506 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.604276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45565 (local address 127.0.0.1:59420)
0808 18:33:13.604660 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.604706 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:13.605074 (+ 368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:13.605485 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:13.605491 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:13.605993 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:13.606360 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:13.606375 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.607441 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.607443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:13.608539 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:13.608550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.609154 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.609157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:13.609295 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.609964 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:13.609986 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:13.611829 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:13.613990 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:13.614001 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:13.614010 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:13.614474 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:13.614834 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:13.614837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:13.614840 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:13.615027 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:13.615556 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:13.615561 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:13.615869 (+ 308us) client_negotiation.cc:769] Sending connection context
0808 18:33:13.616664 (+ 795us) client_negotiation.cc:240] Negotiation successful
0808 18:33:13.616859 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":27,"thread_start_us":152,"threads_started":1}
I20260808 18:33:13.617511 22457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.603773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40762 (local address 127.29.78.188:33261)
0808 18:33:13.604250 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.604271 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.604312 (+ 41us) server_negotiation.cc:407] Connection header received
0808 18:33:13.605054 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.605071 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.605296 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.605457 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.606361 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.607605 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.608754 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.609272 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.609363 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.612510 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:13.612520 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:13.612533 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:13.612558 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:13.614565 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.615063 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.615069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.615071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.615164 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.615509 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.615513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.615514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.615846 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:13.616070 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:13.616319 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 18:33:13.616643 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":309,"spinlock_wait_cycles":896,"thread_start_us":130,"threads_started":1}
I20260808 18:33:13.619736 22454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:33:13.619768 22369 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } attempt: 3
Aug 08 18:33:13 dist-test-slave-gp6n krb5kdc[22194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213993, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
Aug 08 18:33:13 dist-test-slave-gp6n krb5kdc[22194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213993, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
I20260808 18:33:13.626199 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.614385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33261 (local address 127.0.0.1:40778)
0808 18:33:13.614609 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.614643 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:13.614716 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:13.615260 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:13.615263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:13.615437 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:13.615559 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:13.615568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.616831 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.616835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:13.617629 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:13.617640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.618457 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.618460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:13.618534 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.619025 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:13.619039 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:13.621079 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:13.623812 (+ 2733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:13.623816 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:13.623819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:13.624070 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:13.624392 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:13.624398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:13.624400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:13.624503 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:13.625660 (+ 1157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:13.625664 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:13.625781 (+ 117us) client_negotiation.cc:769] Sending connection context
0808 18:33:13.625872 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 18:33:13.626002 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
Aug 08 18:33:13 dist-test-slave-gp6n krb5kdc[22194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213993, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
I20260808 18:33:13.626839 22459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.614587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40778 (local address 127.29.78.188:33261)
0808 18:33:13.615034 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.615042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.615056 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:33:13.615092 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.615096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.615147 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.615241 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.615678 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.616411 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.617994 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.618304 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.618442 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.621207 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:13.621274 (+ 67us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:13.621280 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:13.621310 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:13.623680 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.624192 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.624195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.624196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.624288 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.625193 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.625197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.625199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.625520 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:13.626398 (+ 878us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:13.626526 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 18:33:13.626643 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"spinlock_wait_cycles":33152,"thread_start_us":107,"threads_started":1}
Aug 08 18:33:13 dist-test-slave-gp6n krb5kdc[22194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:33:13.630592 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.620198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33261 (local address 127.0.0.1:40782)
0808 18:33:13.620683 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.620697 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:13.620787 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:13.621574 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:13.621577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:13.621769 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:13.621855 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:13.621866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.622957 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.622960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:13.623772 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:13.623782 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.624372 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.624375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:13.624461 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.624897 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:13.624908 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:13.626695 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:13.628839 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:13.628842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:13.628844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:13.629175 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:13.629502 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:13.629504 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:13.629506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:13.629594 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:13.630032 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:13.630035 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:13.630129 (+ 94us) client_negotiation.cc:769] Sending connection context
0808 18:33:13.630286 (+ 157us) client_negotiation.cc:240] Negotiation successful
0808 18:33:13.630445 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383}
I20260808 18:33:13.631124 22457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.620441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40782 (local address 127.29.78.188:33261)
0808 18:33:13.620563 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.620568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.621399 (+ 831us) server_negotiation.cc:407] Connection header received
0808 18:33:13.621422 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.621425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.621456 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.621969 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.622051 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.622830 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.623915 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.624250 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.624663 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.626845 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:13.626849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:13.626852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:13.626881 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:13.628701 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.629297 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.629301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.629318 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.629397 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.629703 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.629705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.629707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.629896 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:13.630771 (+ 875us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:13.630846 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 18:33:13.630963 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:33:13.631446 22285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e: Bootstrap starting.
I20260808 18:33:13.631587 22374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.620702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55722 (local address 127.29.78.189:43685)
0808 18:33:13.620797 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.620800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.621305 (+ 505us) server_negotiation.cc:407] Connection header received
0808 18:33:13.621399 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.621403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.621452 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.621523 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.622034 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.622836 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.623907 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.624263 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.624347 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.627639 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:13.627643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:13.627646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:13.627672 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:13.629698 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.630191 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.630194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.630196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.630283 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.630641 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.630645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.630647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.630858 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:13.630935 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:13.631300 (+ 365us) server_negotiation.cc:299] Negotiation successful
0808 18:33:13.631422 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 18:33:13.631989 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.620462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:43685 (local address 127.0.0.1:55722)
0808 18:33:13.621216 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.621227 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:13.621322 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:13.621637 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:13.621640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:13.621795 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:13.621896 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:13.621906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.622961 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.622964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:13.623774 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:13.623782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.624488 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.624492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:13.624612 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.625205 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:13.625219 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:13.627448 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:13.629821 (+ 2373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:13.629824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:13.629826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:13.630070 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:13.630431 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:13.630434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:13.630435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:13.630530 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:13.631012 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:13.631016 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:13.631105 (+ 89us) client_negotiation.cc:769] Sending connection context
0808 18:33:13.631709 (+ 604us) client_negotiation.cc:240] Negotiation successful
0808 18:33:13.631846 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698}
I20260808 18:33:13.632632 22454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:33:13.633953 22369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d634286b94b4cc3a69afcdad3c4aa1d: Bootstrap starting.
I20260808 18:33:13.634238 22285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:13.635075 22285 log.cc:824] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:13.636255 22369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d634286b94b4cc3a69afcdad3c4aa1d: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:13.637014 22369 log.cc:824] T 00000000000000000000000000000000 P 4d634286b94b4cc3a69afcdad3c4aa1d: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:13.637171 22285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e: No bootstrap required, opened a new log
Aug 08 18:33:13 dist-test-slave-gp6n krb5kdc[22194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213993, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
I20260808 18:33:13.639240 22289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.620705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59426 (local address 127.29.78.190:45565)
0808 18:33:13.620832 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.620836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.620847 (+ 11us) server_negotiation.cc:407] Connection header received
0808 18:33:13.620876 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.620879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.620920 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.621007 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.621834 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.622645 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.623867 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.624227 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.624299 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.627803 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:13.627808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:13.627811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:13.627838 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:13.629692 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.630191 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.630194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.630196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.630290 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.630967 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.630984 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.630986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.637912 (+ 6926us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:13.638070 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:13.638893 (+ 823us) server_negotiation.cc:299] Negotiation successful
0808 18:33:13.639047 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 18:33:13.639609 22369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d634286b94b4cc3a69afcdad3c4aa1d: No bootstrap required, opened a new log
I20260808 18:33:13.641180 22285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401f60a39d854f4c82072cf622d8a48e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45565 } } peers { permanent_uuid: "4d634286b94b4cc3a69afcdad3c4aa1d" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } } peers { permanent_uuid: "412f573344b84480b888c1acf710fe46" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } }
I20260808 18:33:13.641417 22285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401f60a39d854f4c82072cf622d8a48e, State: Initialized, Role: FOLLOWER
I20260808 18:33:13.641731 22457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.633238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40786 (local address 127.29.78.188:33261)
0808 18:33:13.633329 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.633333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.633345 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:33:13.633375 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.633378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.633413 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.633491 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.633893 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.634626 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.635656 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.635950 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.636067 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.638002 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:13.638006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:13.638008 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:13.638029 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:13.639726 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.640476 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.640479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.640480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.640558 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.640854 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.640857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.640858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.641052 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:13.641127 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:13.641449 (+ 322us) server_negotiation.cc:299] Negotiation successful
0808 18:33:13.641569 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":40}
I20260808 18:33:13.641780 22285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401f60a39d854f4c82072cf622d8a48e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45565 } } peers { permanent_uuid: "4d634286b94b4cc3a69afcdad3c4aa1d" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } } peers { permanent_uuid: "412f573344b84480b888c1acf710fe46" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } }
I20260808 18:33:13.642189 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.633056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33261 (local address 127.0.0.1:40786)
0808 18:33:13.633185 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:13.633200 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:13.633272 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:13.633528 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:13.633531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:13.633687 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:13.633770 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:13.633776 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.634741 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.634745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:13.635522 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:13.635533 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.636064 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.636067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:13.636170 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.636614 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:13.636627 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:13.637872 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:13.639841 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:13.639843 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:13.639845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:13.640367 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:13.640665 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:13.640671 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:13.640675 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:13.640761 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:13.641186 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:13.641189 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:13.641273 (+ 84us) client_negotiation.cc:769] Sending connection context
0808 18:33:13.641864 (+ 591us) client_negotiation.cc:240] Negotiation successful
0808 18:33:13.641993 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":19}
I20260808 18:33:13.643178 22285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:13.644276 22369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d634286b94b4cc3a69afcdad3c4aa1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401f60a39d854f4c82072cf622d8a48e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45565 } } peers { permanent_uuid: "4d634286b94b4cc3a69afcdad3c4aa1d" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } } peers { permanent_uuid: "412f573344b84480b888c1acf710fe46" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } }
I20260808 18:33:13.644570 22369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d634286b94b4cc3a69afcdad3c4aa1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d634286b94b4cc3a69afcdad3c4aa1d, State: Initialized, Role: FOLLOWER
I20260808 18:33:13.645025 22454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412f573344b84480b888c1acf710fe46: Bootstrap starting.
I20260808 18:33:13.644966 22369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d634286b94b4cc3a69afcdad3c4aa1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401f60a39d854f4c82072cf622d8a48e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45565 } } peers { permanent_uuid: "4d634286b94b4cc3a69afcdad3c4aa1d" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } } peers { permanent_uuid: "412f573344b84480b888c1acf710fe46" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } }
I20260808 18:33:13.646466 22369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d634286b94b4cc3a69afcdad3c4aa1d [sys.catalog]: configured and running, proceeding with master startup.
Aug 08 18:33:13 dist-test-slave-gp6n krb5kdc[22194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
I20260808 18:33:13.647903 22454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 412f573344b84480b888c1acf710fe46: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:13.648808 22454 log.cc:824] T 00000000000000000000000000000000 P 412f573344b84480b888c1acf710fe46: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:13.652231 22374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.641093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55734 (local address 127.29.78.189:43685)
0808 18:33:13.641202 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.641206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.641359 (+ 153us) server_negotiation.cc:407] Connection header received
0808 18:33:13.641535 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.641541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.641585 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.641633 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.643080 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.643805 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.644936 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.645236 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.645315 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.648393 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:13.648397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:13.648400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:13.648424 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:13.650226 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.650781 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.650785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.650786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.650880 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.651205 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.651208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.651210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.651414 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:13.651598 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:13.651970 (+ 372us) server_negotiation.cc:299] Negotiation successful
0808 18:33:13.652094 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 18:33:13.648136 22209 master_runner.cc:186] Error getting master registration for 127.29.78.189:43685: OK, instance_id { permanent_uuid: "4d634286b94b4cc3a69afcdad3c4aa1d" instance_seqno: 1786213993368187 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 18:33:13.652431 22209 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 18:33:13.652539 22488 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d634286b94b4cc3a69afcdad3c4aa1d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:33:13.652611 22488 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:33:13.653517 22474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d634286b94b4cc3a69afcdad3c4aa1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401f60a39d854f4c82072cf622d8a48e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45565 } } peers { permanent_uuid: "4d634286b94b4cc3a69afcdad3c4aa1d" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } } peers { permanent_uuid: "412f573344b84480b888c1acf710fe46" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } } }
I20260808 18:33:13.653882 22474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d634286b94b4cc3a69afcdad3c4aa1d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:33:13.654139 22462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401f60a39d854f4c82072cf622d8a48e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45565 } } peers { permanent_uuid: "4d634286b94b4cc3a69afcdad3c4aa1d" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } } peers { permanent_uuid: "412f573344b84480b888c1acf710fe46" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } } }
I20260808 18:33:13.654251 22462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e [sys.catalog]: This master's current role is: FOLLOWER
W20260808 18:33:13.655493 22473 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:33:13.655568 22473 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:33:13.654672 22292 master_runner.cc:186] Error getting master registration for 127.29.78.188:33261: OK, instance_id { permanent_uuid: "412f573344b84480b888c1acf710fe46" instance_seqno: 1786213993593218 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 18:33:13.658545 22292 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 18:33:13.659304 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:33:13.660362 22454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412f573344b84480b888c1acf710fe46: No bootstrap required, opened a new log
I20260808 18:33:13.660769 22256 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "401f60a39d854f4c82072cf622d8a48e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59426
I20260808 18:33:13.661075 22256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 18:33:13.661188 22256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:13.662688 22256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401f60a39d854f4c82072cf622d8a48e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45565 } } peers { permanent_uuid: "4d634286b94b4cc3a69afcdad3c4aa1d" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } } peers { permanent_uuid: "412f573344b84480b888c1acf710fe46" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } }
I20260808 18:33:13.663691 22256 leader_election.cc:288] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e [CANDIDATE]: Term 1 election: Requested vote from peers 4d634286b94b4cc3a69afcdad3c4aa1d (127.29.78.189:43685), 412f573344b84480b888c1acf710fe46 (127.29.78.188:33261)
I20260808 18:33:13.664515 22454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 412f573344b84480b888c1acf710fe46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401f60a39d854f4c82072cf622d8a48e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45565 } } peers { permanent_uuid: "4d634286b94b4cc3a69afcdad3c4aa1d" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } } peers { permanent_uuid: "412f573344b84480b888c1acf710fe46" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } }
I20260808 18:33:13.664762 22454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 412f573344b84480b888c1acf710fe46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412f573344b84480b888c1acf710fe46, State: Initialized, Role: FOLLOWER
I20260808 18:33:13.665037 22340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "401f60a39d854f4c82072cf622d8a48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4d634286b94b4cc3a69afcdad3c4aa1d"
I20260808 18:33:13.665105 22454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 412f573344b84480b888c1acf710fe46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401f60a39d854f4c82072cf622d8a48e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45565 } } peers { permanent_uuid: "4d634286b94b4cc3a69afcdad3c4aa1d" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } } peers { permanent_uuid: "412f573344b84480b888c1acf710fe46" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } }
I20260808 18:33:13.665256 22340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d634286b94b4cc3a69afcdad3c4aa1d [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:13.666603 22340 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4d634286b94b4cc3a69afcdad3c4aa1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401f60a39d854f4c82072cf622d8a48e in term 1.
I20260808 18:33:13.666662 22491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412f573344b84480b888c1acf710fe46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401f60a39d854f4c82072cf622d8a48e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45565 } } peers { permanent_uuid: "4d634286b94b4cc3a69afcdad3c4aa1d" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } } peers { permanent_uuid: "412f573344b84480b888c1acf710fe46" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } } }
I20260808 18:33:13.666810 22491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412f573344b84480b888c1acf710fe46 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:33:13.666662 22454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 412f573344b84480b888c1acf710fe46 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:13.667055 22226 leader_election.cc:302] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401f60a39d854f4c82072cf622d8a48e, 4d634286b94b4cc3a69afcdad3c4aa1d; no voters:
I20260808 18:33:13.667029 22426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "401f60a39d854f4c82072cf622d8a48e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "412f573344b84480b888c1acf710fe46"
I20260808 18:33:13.667227 22426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 412f573344b84480b888c1acf710fe46 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:13.667338 22462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:13.667547 22462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e [term 1 LEADER]: Becoming Leader. State: Replica: 401f60a39d854f4c82072cf622d8a48e, State: Running, Role: LEADER
I20260808 18:33:13.667893 22462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401f60a39d854f4c82072cf622d8a48e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45565 } } peers { permanent_uuid: "4d634286b94b4cc3a69afcdad3c4aa1d" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } } peers { permanent_uuid: "412f573344b84480b888c1acf710fe46" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } }
I20260808 18:33:13.668987 22426 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 412f573344b84480b888c1acf710fe46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401f60a39d854f4c82072cf622d8a48e in term 1.
I20260808 18:33:13.670101 22462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 401f60a39d854f4c82072cf622d8a48e. Latest consensus state: current_term: 1 leader_uuid: "401f60a39d854f4c82072cf622d8a48e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401f60a39d854f4c82072cf622d8a48e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45565 } } peers { permanent_uuid: "4d634286b94b4cc3a69afcdad3c4aa1d" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } } peers { permanent_uuid: "412f573344b84480b888c1acf710fe46" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } } }
I20260808 18:33:13.670221 22462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:13.672026 22508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:13.672941 22508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260808 18:33:13.680104 22511 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 412f573344b84480b888c1acf710fe46: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:33:13.680181 22511 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:33:13.681499 22426 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 412f573344b84480b888c1acf710fe46 [term 1 FOLLOWER]: Refusing update from remote peer 401f60a39d854f4c82072cf622d8a48e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:13.682147 22340 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d634286b94b4cc3a69afcdad3c4aa1d [term 1 FOLLOWER]: Refusing update from remote peer 401f60a39d854f4c82072cf622d8a48e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:13.682844 22462 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e [LEADER]: Connected to new peer: Peer: permanent_uuid: "412f573344b84480b888c1acf710fe46" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:33:13.683219 22462 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d634286b94b4cc3a69afcdad3c4aa1d" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:33:13.683961 22289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.671710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59428 (local address 127.29.78.190:45565)
0808 18:33:13.671958 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.671963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.674181 (+ 2218us) server_negotiation.cc:407] Connection header received
0808 18:33:13.674383 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.674386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.674497 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.674551 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.675628 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.676360 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.677577 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.677909 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.677995 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.679852 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:13.679856 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:13.679859 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:13.679884 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:13.681813 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.682485 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.682489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.682491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.682575 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.682863 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.682867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.682869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.683094 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:13.683204 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:13.683684 (+ 480us) server_negotiation.cc:299] Negotiation successful
0808 18:33:13.683798 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260808 18:33:13.689349 22491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412f573344b84480b888c1acf710fe46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 401f60a39d854f4c82072cf622d8a48e. Latest consensus state: current_term: 1 leader_uuid: "401f60a39d854f4c82072cf622d8a48e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401f60a39d854f4c82072cf622d8a48e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45565 } } peers { permanent_uuid: "4d634286b94b4cc3a69afcdad3c4aa1d" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } } peers { permanent_uuid: "412f573344b84480b888c1acf710fe46" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } } }
I20260808 18:33:13.689471 22491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412f573344b84480b888c1acf710fe46 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:33:13.690685 22474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d634286b94b4cc3a69afcdad3c4aa1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 401f60a39d854f4c82072cf622d8a48e. Latest consensus state: current_term: 1 leader_uuid: "401f60a39d854f4c82072cf622d8a48e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401f60a39d854f4c82072cf622d8a48e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45565 } } peers { permanent_uuid: "4d634286b94b4cc3a69afcdad3c4aa1d" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } } peers { permanent_uuid: "412f573344b84480b888c1acf710fe46" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } } }
I20260808 18:33:13.690802 22474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d634286b94b4cc3a69afcdad3c4aa1d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:33:13.696297 22462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "401f60a39d854f4c82072cf622d8a48e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401f60a39d854f4c82072cf622d8a48e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45565 } } peers { permanent_uuid: "4d634286b94b4cc3a69afcdad3c4aa1d" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } } peers { permanent_uuid: "412f573344b84480b888c1acf710fe46" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } } }
I20260808 18:33:13.696426 22462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:13.696800 22462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "401f60a39d854f4c82072cf622d8a48e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401f60a39d854f4c82072cf622d8a48e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45565 } } peers { permanent_uuid: "4d634286b94b4cc3a69afcdad3c4aa1d" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } } peers { permanent_uuid: "412f573344b84480b888c1acf710fe46" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } } }
I20260808 18:33:13.696892 22462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:13.697000 22474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d634286b94b4cc3a69afcdad3c4aa1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "401f60a39d854f4c82072cf622d8a48e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401f60a39d854f4c82072cf622d8a48e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45565 } } peers { permanent_uuid: "4d634286b94b4cc3a69afcdad3c4aa1d" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } } peers { permanent_uuid: "412f573344b84480b888c1acf710fe46" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } } }
I20260808 18:33:13.697116 22474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d634286b94b4cc3a69afcdad3c4aa1d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:33:13.697227 22516 mvcc.cc:205] Tried to move back new op lower bound from 7316332518099980288 to 7316332518067724288. Current Snapshot: MvccSnapshot[applied={T|T < 7316332518099980288}]
I20260808 18:33:13.698170 22491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412f573344b84480b888c1acf710fe46 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "401f60a39d854f4c82072cf622d8a48e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401f60a39d854f4c82072cf622d8a48e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45565 } } peers { permanent_uuid: "4d634286b94b4cc3a69afcdad3c4aa1d" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 43685 } } peers { permanent_uuid: "412f573344b84480b888c1acf710fe46" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33261 } } }
I20260808 18:33:13.698292 22491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412f573344b84480b888c1acf710fe46 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:33:13.699096 22520 mvcc.cc:205] Tried to move back new op lower bound from 7316332518099980288 to 7316332518067724288. Current Snapshot: MvccSnapshot[applied={T|T < 7316332518099980288}]
I20260808 18:33:13.699949 22457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.673603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40788 (local address 127.29.78.188:33261)
0808 18:33:13.673705 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.673709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.687510 (+ 13801us) server_negotiation.cc:407] Connection header received
0808 18:33:13.687557 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.687560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.687604 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.687673 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.688557 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.689353 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.690632 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.690996 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.691071 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.692910 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:13.692915 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:13.692917 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:13.692956 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:13.694690 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.698499 (+ 3809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.698503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.698505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.698597 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.698912 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.698915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.698917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.699138 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:13.699328 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:13.699665 (+ 337us) server_negotiation.cc:299] Negotiation successful
0808 18:33:13.699781 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 18:33:13.704216 22508 catalog_manager.cc:1347] Generated new cluster ID: 67377c17288c4e2eb88e970b5b11f44b
I20260808 18:33:13.704286 22508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:13.707600 22518 mvcc.cc:205] Tried to move back new op lower bound from 7316332518099980288 to 7316332518067724288. Current Snapshot: MvccSnapshot[applied={T|T < 7316332518099980288}]
I20260808 18:33:13.709147 22374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:13.672966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55746 (local address 127.29.78.189:43685)
0808 18:33:13.673079 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.673083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.685530 (+ 12447us) server_negotiation.cc:407] Connection header received
0808 18:33:13.685685 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.685689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.685735 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.685783 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.686642 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.687419 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.701644 (+ 14225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:13.702023 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:13.702097 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:13.704433 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:13.704437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:13.704439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:13.704464 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:13.706129 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.707955 (+ 1826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.707959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.707960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.708037 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:13.708291 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:13.708294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:13.708296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:13.708475 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:13.708578 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:13.708882 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 18:33:13.709003 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:33:13.733359 22508 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:13.734601 22508 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:13.753513 22508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 401f60a39d854f4c82072cf622d8a48e: Generated new TSK 0
I20260808 18:33:13.754379 22508 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 18:33:13.793655 22289 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:13.790569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59442 (local address 127.29.78.190:45565)
0808 18:33:13.790696 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.790701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.792550 (+ 1849us) server_negotiation.cc:407] Connection header received
0808 18:33:13.792751 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.792757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.792806 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.792859 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.793533 (+ 674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59442: BlockingRecv error: recv got EOF from 127.0.0.1:59442 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260808 18:33:13.795099 22374 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:13.791362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55754 (local address 127.29.78.189:43685)
0808 18:33:13.791469 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.791474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.794172 (+ 2698us) server_negotiation.cc:407] Connection header received
0808 18:33:13.794342 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.794346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.794394 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.794469 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.794989 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55754: BlockingRecv error: recv got EOF from 127.0.0.1:55754 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260808 18:33:13.796326 22457 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:13.792024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40798 (local address 127.29.78.188:33261)
0808 18:33:13.792154 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.792159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.795561 (+ 3402us) server_negotiation.cc:407] Connection header received
0808 18:33:13.795684 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.795687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.795738 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.795827 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.796210 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40798: BlockingRecv error: recv got EOF from 127.0.0.1:40798 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260808 18:33:13.800168 22374 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:13.798154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55756 (local address 127.29.78.189:43685)
0808 18:33:13.798258 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.798262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.799238 (+ 976us) server_negotiation.cc:407] Connection header received
0808 18:33:13.799447 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.799450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.799536 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.799592 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.800080 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55756: BlockingRecv error: recv got EOF from 127.0.0.1:55756 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260808 18:33:13.805158 22374 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:13.803761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55768 (local address 127.29.78.189:43685)
0808 18:33:13.803855 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.803858 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.804314 (+ 456us) server_negotiation.cc:407] Connection header received
0808 18:33:13.804485 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.804488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.804537 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.804586 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.805064 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55768: BlockingRecv error: recv got EOF from 127.0.0.1:55768 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260808 18:33:13.812323 22374 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:13.810825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55770 (local address 127.29.78.189:43685)
0808 18:33:13.810927 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.810931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.811416 (+ 485us) server_negotiation.cc:407] Connection header received
0808 18:33:13.811610 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.811613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.811663 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.811713 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.812230 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55770: BlockingRecv error: recv got EOF from 127.0.0.1:55770 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260808 18:33:13.820261 22374 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:13.818592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55786 (local address 127.29.78.189:43685)
0808 18:33:13.818729 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.818733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.819246 (+ 513us) server_negotiation.cc:407] Connection header received
0808 18:33:13.819445 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.819448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.819497 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.819547 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.820138 (+ 591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55786: BlockingRecv error: recv got EOF from 127.0.0.1:55786 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260808 18:33:13.832188 22374 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:13.830642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55800 (local address 127.29.78.189:43685)
0808 18:33:13.830743 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.830748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.831237 (+ 489us) server_negotiation.cc:407] Connection header received
0808 18:33:13.831437 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.831440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.831493 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.831549 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.832088 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55800: BlockingRecv error: recv got EOF from 127.0.0.1:55800 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260808 18:33:13.852296 22374 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:13.850676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55812 (local address 127.29.78.189:43685)
0808 18:33:13.850790 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.850794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.851336 (+ 542us) server_negotiation.cc:407] Connection header received
0808 18:33:13.851538 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.851542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.851593 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.851643 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.852182 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55812: BlockingRecv error: recv got EOF from 127.0.0.1:55812 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260808 18:33:13.888250 22374 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:13.886971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55824 (local address 127.29.78.189:43685)
0808 18:33:13.887132 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.887137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.887319 (+ 182us) server_negotiation.cc:407] Connection header received
0808 18:33:13.887461 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.887463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.887513 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.887587 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.888125 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55824: BlockingRecv error: recv got EOF from 127.0.0.1:55824 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260808 18:33:13.955806 22374 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:13.954568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55840 (local address 127.29.78.189:43685)
0808 18:33:13.954725 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:13.954730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:13.954925 (+ 195us) server_negotiation.cc:407] Connection header received
0808 18:33:13.955063 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:13.955065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:13.955116 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:13.955191 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:13.955657 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55840: BlockingRecv error: recv got EOF from 127.0.0.1:55840 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260808 18:33:14.087324 22374 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:14.085976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.29.78.189:43685)
0808 18:33:14.086148 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:14.086153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:14.086311 (+ 158us) server_negotiation.cc:407] Connection header received
0808 18:33:14.086539 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:14.086542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:14.086592 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:14.086691 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:14.087201 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55854: BlockingRecv error: recv got EOF from 127.0.0.1:55854 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260808 18:33:14.346779 22374 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:14.345626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55868 (local address 127.29.78.189:43685)
0808 18:33:14.345796 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:14.345800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:14.345861 (+ 61us) server_negotiation.cc:407] Connection header received
0808 18:33:14.345996 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:14.345999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:14.346047 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:14.346121 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:14.346624 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55868: BlockingRecv error: recv got EOF from 127.0.0.1:55868 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 18:33:14.653927 22488 catalog_manager.cc:1259] Loaded cluster ID: 67377c17288c4e2eb88e970b5b11f44b
I20260808 18:33:14.654022 22488 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4d634286b94b4cc3a69afcdad3c4aa1d: loading cluster ID for follower catalog manager: success
I20260808 18:33:14.655676 22488 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4d634286b94b4cc3a69afcdad3c4aa1d: acquiring CA information for follower catalog manager: success
I20260808 18:33:14.656857 22488 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4d634286b94b4cc3a69afcdad3c4aa1d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 18:33:14.681319 22511 catalog_manager.cc:1259] Loaded cluster ID: 67377c17288c4e2eb88e970b5b11f44b
I20260808 18:33:14.681423 22511 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 412f573344b84480b888c1acf710fe46: loading cluster ID for follower catalog manager: success
I20260808 18:33:14.683081 22511 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 412f573344b84480b888c1acf710fe46: acquiring CA information for follower catalog manager: success
I20260808 18:33:14.684213 22511 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 412f573344b84480b888c1acf710fe46: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 18:33:14.864960 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:14.861103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55882 (local address 127.29.78.189:43685)
0808 18:33:14.861382 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:14.861386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:14.861403 (+ 17us) server_negotiation.cc:407] Connection header received
0808 18:33:14.861542 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:14.861544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:14.861674 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:14.861776 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:14.862032 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:14.862634 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:14.863488 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:14.863730 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:14.863813 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:14.864143 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:14.864584 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:14.864692 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:14.864740 (+ 48us) server_negotiation.cc:299] Negotiation successful
0808 18:33:14.864792 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
I20260808 18:33:14.871974 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:14.868337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40800 (local address 127.29.78.188:33261)
0808 18:33:14.868594 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:14.868597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:14.868611 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:33:14.868730 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:14.868732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:14.868864 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:14.868959 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:14.869125 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:14.869625 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:14.870517 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:14.870789 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:14.870874 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:14.871159 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:14.871562 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:14.871681 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:14.871717 (+ 36us) server_negotiation.cc:299] Negotiation successful
0808 18:33:14.871793 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":99,"threads_started":1}
I20260808 18:33:14.878706 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:14.875539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55898 (local address 127.29.78.189:43685)
0808 18:33:14.875639 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:14.875642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:14.875652 (+ 10us) server_negotiation.cc:407] Connection header received
0808 18:33:14.875689 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:14.875691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:14.875757 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:14.875835 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:14.876088 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:14.876655 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:14.877667 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:14.877874 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:14.877949 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:14.878220 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:14.878294 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:14.878372 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:14.878521 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 18:33:14.878574 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 18:33:14.879608 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:14.874958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.29.78.190:45565)
0808 18:33:14.875382 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:14.875386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:14.875402 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:33:14.875508 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:14.875512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:14.875685 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:14.875826 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:14.876206 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:14.877049 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:14.878030 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:14.878315 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:14.878437 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:14.878839 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:14.879164 (+ 325us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:14.879273 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:14.879384 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 18:33:14.879441 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":70,"threads_started":1}
I20260808 18:33:14.880170 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:14.875390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40814 (local address 127.29.78.188:33261)
0808 18:33:14.875514 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:14.875519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:14.875609 (+ 90us) server_negotiation.cc:407] Connection header received
0808 18:33:14.875774 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:14.875780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:14.875839 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:14.875901 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:14.876280 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:14.877047 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:14.879022 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:14.879298 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:14.879376 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:14.879748 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:14.879822 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:14.879909 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:14.880009 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 18:33:14.880058 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 18:33:14.882673 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 22209
I20260808 18:33:14.892957 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 22292
I20260808 18:33:14.900549 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 22377
2026-08-08T18:33:14Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3204 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:33:14 dist-test-slave-gp6n krb5kdc[22625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:33:14 dist-test-slave-gp6n krb5kdc[22625](info): set up 2 sockets
Aug 08 18:33:14 dist-test-slave-gp6n krb5kdc[22625](info): commencing operation
krb5kdc: starting...
W20260808 18:33:16.043354 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/test-user.keytab.
Aug 08 18:33:16 dist-test-slave-gp6n krb5kdc[22625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:33:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260808 18:33:16.174320 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:41171
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:41171,127.29.78.189:36349,127.29.78.188:43725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:16.287997 22640 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 18:33:16.288327 22640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:16.288404 22640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:16.288486 22640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:16.292440 22640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:16.292523 22640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:16.292562 22640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:16.292599 22640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:16.292635 22640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:16.297721 22640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:41171,127.29.78.189:36349,127.29.78.188:43725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:41171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:16.298768 22640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:16.299775 22640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:16.309170 22650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:16.309180 22647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:16.309230 22645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:16.309194 22640 server_base.cc:1034] running on GCE node
I20260808 18:33:16.310269 22640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:16.311467 22640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:16.312646 22640 hybrid_clock.cc:648] HybridClock initialized: now 1786213996312617 us; error 45 us; skew 500 ppm
Aug 08 18:33:16 dist-test-slave-gp6n krb5kdc[22625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213996, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:16.315562 22640 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260808 18:33:16.316427 22640 webserver.cc:460] Webserver started at http://127.29.78.190:44303/ using document root <none> and password file <none>
I20260808 18:33:16.316993 22640 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:16.317057 22640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:16.317298 22640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:16.319412 22640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-0/data/instance:
uuid: "855228b4b299464eb7d9b88230cd55d3"
format_stamp: "Formatted at 2026-08-08 18:33:16 on dist-test-slave-gp6n"
server_key: "587699da3c5911a019c9b4c3492c9e8d"
server_key_iv: "968a5c1f986efb81ccfe856bfeae2363"
server_key_version: "encryptionkey@0"
I20260808 18:33:16.319993 22640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-0/wal/instance:
uuid: "855228b4b299464eb7d9b88230cd55d3"
format_stamp: "Formatted at 2026-08-08 18:33:16 on dist-test-slave-gp6n"
server_key: "587699da3c5911a019c9b4c3492c9e8d"
server_key_iv: "968a5c1f986efb81ccfe856bfeae2363"
server_key_version: "encryptionkey@0"
I20260808 18:33:16.323789 22640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:33:16.326114 22657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:16.327553 22640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:33:16.327694 22640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-0/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-0/wal
uuid: "855228b4b299464eb7d9b88230cd55d3"
format_stamp: "Formatted at 2026-08-08 18:33:16 on dist-test-slave-gp6n"
server_key: "587699da3c5911a019c9b4c3492c9e8d"
server_key_iv: "968a5c1f986efb81ccfe856bfeae2363"
server_key_version: "encryptionkey@0"
I20260808 18:33:16.327804 22640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:16.357565 22640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:16.360620 22640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:16.360836 22640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:16.368443 22640 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:41171
I20260808 18:33:16.368758 22717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:41171 every 8 connection(s)
I20260808 18:33:16.369438 22640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:16.371057 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 22640
I20260808 18:33:16.371199 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-0/wal/instance
I20260808 18:33:16.371526 30010 external_mini_cluster.cc:1428] Setting key 725cb3f016733b8a33e39ee96306b4a7
I20260808 18:33:16.372694 22718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:16.376768 22718 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41171 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.78.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.189@KRBTEST.COM" created.
Aug 08 18:33:16 dist-test-slave-gp6n krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213996, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:33:16.392999 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.379239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:41171 (local address 127.0.0.1:36100)
0808 18:33:16.379763 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.379804 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:16.380708 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:16.381421 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:16.381426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:16.381854 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:16.382106 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:16.382118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.383300 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.383304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:16.384103 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:16.384112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.384754 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.384758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:16.384924 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.385602 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:16.385640 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:16.387456 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:16.389968 (+ 2512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:16.389976 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:16.389991 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:16.390224 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:16.390624 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:16.390626 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:16.390627 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:16.390740 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:16.391560 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:16.391565 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:16.391692 (+ 127us) client_negotiation.cc:769] Sending connection context
0808 18:33:16.391855 (+ 163us) client_negotiation.cc:240] Negotiation successful
0808 18:33:16.392146 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":54,"thread_start_us":163,"threads_started":1}
I20260808 18:33:16.393010 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.380259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36100 (local address 127.29.78.190:41171)
0808 18:33:16.380572 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:16.380583 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:16.380627 (+ 44us) server_negotiation.cc:407] Connection header received
0808 18:33:16.381047 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:16.381051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:16.381238 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:16.381423 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:16.382255 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.383174 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.384225 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.384603 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.384836 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.387615 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:16.387625 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:16.387642 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:16.387675 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:16.389849 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.390340 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.390348 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.390352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.390528 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.390837 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.390841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.390842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.391409 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:16.391566 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:16.391999 (+ 433us) server_negotiation.cc:299] Negotiation successful
0808 18:33:16.392222 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":206,"thread_start_us":97,"threads_started":1}
I20260808 18:33:16.395751 22718 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:33:16.396507 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.396263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:36349 (local address 127.0.0.1:54636)
0808 18:33:16.396374 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.396445 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 18:33:16.396714 22660 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.78.189:36349: connect: Connection refused (error 111)
I20260808 18:33:16.397667 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.397389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:36349 (local address 127.0.0.1:54648)
0808 18:33:16.397568 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.397614 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260808 18:33:16.397804 22718 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.189:36349: Network error: Client connection negotiation failed: client connection to 127.29.78.189:36349: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.189.keytab.
Entry for principal kudu/127.29.78.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.189.keytab.
I20260808 18:33:16.402580 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.189:36349
--webserver_interface=127.29.78.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.189.keytab
--principal=kudu/127.29.78.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:41171,127.29.78.189:36349,127.29.78.188:43725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:33:16.447096 22718 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } attempt: 1
I20260808 18:33:16.448125 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.447810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:36349 (local address 127.0.0.1:54654)
0808 18:33:16.447976 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.448050 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 18:33:16.449038 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.448785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:36349 (local address 127.0.0.1:54666)
0808 18:33:16.448915 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.448975 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260808 18:33:16.449237 22718 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.189:36349: Network error: Client connection negotiation failed: client connection to 127.29.78.189:36349: connect: Connection refused (error 111)
W20260808 18:33:16.512591 22726 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 18:33:16.512848 22726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:16.512920 22726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:16.512993 22726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:16.516979 22726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:16.517030 22726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:16.517074 22726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:16.517113 22726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:16.517148 22726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:16.517422 22718 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } attempt: 2
I20260808 18:33:16.518250 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.517938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:36349 (local address 127.0.0.1:54674)
0808 18:33:16.518098 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.518170 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260808 18:33:16.519060 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.518844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:36349 (local address 127.0.0.1:54690)
0808 18:33:16.518951 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.518998 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 18:33:16.519248 22718 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.189:36349: Network error: Client connection negotiation failed: client connection to 127.29.78.189:36349: connect: Connection refused (error 111)
I20260808 18:33:16.522717 22726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:41171,127.29.78.189:36349,127.29.78.188:43725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.189.keytab
--principal=kudu/127.29.78.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.189:36349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.78.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:16.523787 22726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:16.524808 22726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:16.533233 22732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:16.533222 22731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:16.533221 22734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:16.533298 22726 server_base.cc:1034] running on GCE node
I20260808 18:33:16.534047 22726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:16.535154 22726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:16.536343 22726 hybrid_clock.cc:648] HybridClock initialized: now 1786213996536325 us; error 30 us; skew 500 ppm
Aug 08 18:33:16 dist-test-slave-gp6n krb5kdc[22625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213996, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:16.539105 22726 init.cc:377] Logged in from keytab as kudu/127.29.78.189@KRBTEST.COM (short username kudu)
I20260808 18:33:16.539875 22726 webserver.cc:460] Webserver started at http://127.29.78.189:35183/ using document root <none> and password file <none>
I20260808 18:33:16.540362 22726 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:16.540419 22726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:16.540580 22726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:16.542526 22726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-1/data/instance:
uuid: "4c92a480dd42460eb17cec101e5e2dfb"
format_stamp: "Formatted at 2026-08-08 18:33:16 on dist-test-slave-gp6n"
server_key: "2a2f70d3a25757c8a2ba611affe525dc"
server_key_iv: "e5f1ab5a295285a3922762fc9cb37246"
server_key_version: "encryptionkey@0"
I20260808 18:33:16.543043 22726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-1/wal/instance:
uuid: "4c92a480dd42460eb17cec101e5e2dfb"
format_stamp: "Formatted at 2026-08-08 18:33:16 on dist-test-slave-gp6n"
server_key: "2a2f70d3a25757c8a2ba611affe525dc"
server_key_iv: "e5f1ab5a295285a3922762fc9cb37246"
server_key_version: "encryptionkey@0"
I20260808 18:33:16.546619 22726 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260808 18:33:16.548998 22743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:16.550274 22726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:33:16.550501 22726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-1/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-1/wal
uuid: "4c92a480dd42460eb17cec101e5e2dfb"
format_stamp: "Formatted at 2026-08-08 18:33:16 on dist-test-slave-gp6n"
server_key: "2a2f70d3a25757c8a2ba611affe525dc"
server_key_iv: "e5f1ab5a295285a3922762fc9cb37246"
server_key_version: "encryptionkey@0"
I20260808 18:33:16.550606 22726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:16.569042 22726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:16.572672 22726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:16.572896 22726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:16.580515 22726 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.189:36349
I20260808 18:33:16.580825 22800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.189:36349 every 8 connection(s)
I20260808 18:33:16.581542 22726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-1/data/info.pb
I20260808 18:33:16.584313 22801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:16.587852 22801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41171 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:33:16.589388 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 22726
I20260808 18:33:16.589582 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-1/wal/instance
I20260808 18:33:16.589887 30010 external_mini_cluster.cc:1428] Setting key 00055af9887d7de288904b30d5cf0ff6
WARNING: no policy specified for kudu/127.29.78.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.188@KRBTEST.COM" created.
Aug 08 18:33:16 dist-test-slave-gp6n krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213996, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:33:16.607635 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.589632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36106 (local address 127.29.78.190:41171)
0808 18:33:16.589770 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:16.589774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:16.590755 (+ 981us) server_negotiation.cc:407] Connection header received
0808 18:33:16.591715 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:16.591722 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:16.591793 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:16.591905 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:16.593541 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.594073 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.595954 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.596195 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.596278 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.603119 (+ 6841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:16.603126 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:16.603132 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:16.603170 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:16.604885 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.605573 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.605576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.605577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.605670 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.606180 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.606183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.606185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.606533 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:16.606653 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:16.607388 (+ 735us) server_negotiation.cc:299] Negotiation successful
0808 18:33:16.607504 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 18:33:16.608880 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.590078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:41171 (local address 127.0.0.1:36106)
0808 18:33:16.590565 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.590620 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:16.591503 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:16.592132 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:16.592144 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:16.592649 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:16.593381 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:16.593404 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.594213 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.594219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:16.595816 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:16.595832 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.596365 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.596371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:16.596620 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.601126 (+ 4506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:16.601143 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:16.602926 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:16.605017 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:16.605026 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:16.605053 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:16.605427 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:16.605770 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:16.605777 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:16.605781 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:16.605974 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:16.606672 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:16.606683 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:16.607026 (+ 343us) client_negotiation.cc:769] Sending connection context
0808 18:33:16.607258 (+ 232us) client_negotiation.cc:240] Negotiation successful
0808 18:33:16.607651 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":133,"threads_started":1}
I20260808 18:33:16.610461 22801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:33:16.610425 22718 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } attempt: 3
Aug 08 18:33:16 dist-test-slave-gp6n krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213996, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
Aug 08 18:33:16 dist-test-slave-gp6n krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213996, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.188.keytab.
Entry for principal kudu/127.29.78.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.188.keytab.
I20260808 18:33:16.621097 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0kHl76/build/debug/bin/kudu
/tmp/dist-test-task0kHl76/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.188:43725
--webserver_interface=127.29.78.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.188.keytab
--principal=kudu/127.29.78.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:41171,127.29.78.189:36349,127.29.78.188:43725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:33:16.622246 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.610821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:36349 (local address 127.0.0.1:54698)
0808 18:33:16.611304 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.611318 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:16.611386 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:16.611889 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:16.611891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:16.612091 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:16.612209 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:16.612218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.613508 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.613513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:16.614766 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:16.614778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.615392 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.615396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:16.615510 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.615915 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:16.615935 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:16.617676 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:16.620611 (+ 2935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:16.620614 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:16.620617 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:16.620822 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:16.621136 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:16.621138 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:16.621140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:16.621239 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:16.621692 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:16.621695 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:16.621796 (+ 101us) client_negotiation.cc:769] Sending connection context
0808 18:33:16.621914 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 18:33:16.622070 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401}
I20260808 18:33:16.622272 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.610970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:36349 (local address 127.0.0.1:54710)
0808 18:33:16.611110 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.611126 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:16.611201 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:16.611997 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:16.612000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:16.612225 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:16.612418 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:16.612426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.613543 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.613548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:16.614169 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:16.614178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.615018 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.615021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:16.615195 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.615734 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:16.615752 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:16.618067 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:16.620281 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:16.620287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:16.620289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:16.620607 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:16.620946 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:16.620949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:16.620950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:16.621078 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:16.621644 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:16.621647 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:16.621745 (+ 98us) client_negotiation.cc:769] Sending connection context
0808 18:33:16.621833 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 18:33:16.622015 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":66}
I20260808 18:33:16.622349 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.610926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54698 (local address 127.29.78.189:36349)
0808 18:33:16.611428 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:16.611442 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:16.611464 (+ 22us) server_negotiation.cc:407] Connection header received
0808 18:33:16.611535 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:16.611541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:16.611746 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:16.611883 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:16.612320 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.613387 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.614919 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.615254 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.615411 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.617820 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:16.617829 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:16.617839 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:16.617861 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:16.620497 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.620924 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.620929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.620931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.621033 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.621363 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.621366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.621367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.621578 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:16.621717 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:16.622027 (+ 310us) server_negotiation.cc:299] Negotiation successful
0808 18:33:16.622170 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":410,"spinlock_wait_cycles":3968,"thread_start_us":157,"threads_started":1}
I20260808 18:33:16.622951 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.611189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54710 (local address 127.29.78.189:36349)
0808 18:33:16.611568 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:16.611576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:16.611587 (+ 11us) server_negotiation.cc:407] Connection header received
0808 18:33:16.611610 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:16.611613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:16.611746 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:16.611883 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:16.612538 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.613399 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.614281 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.614900 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.615023 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.618223 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:16.618225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:16.618227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:16.618246 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:16.620179 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.620728 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.620735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.620738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.620829 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.621177 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.621182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.621184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.621532 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:16.622484 (+ 952us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:16.622564 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 18:33:16.622686 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":241,"server-negotiator.queue_time_us":324,"spinlock_wait_cycles":12416,"thread_start_us":98,"threads_started":1}
I20260808 18:33:16.624964 22718 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:33:16.625079 22801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:33:16.625679 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.625429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43725 (local address 127.0.0.1:51838)
0808 18:33:16.625555 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.625633 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 18:33:16.625808 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.625477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43725 (local address 127.0.0.1:51852)
0808 18:33:16.625661 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.625720 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260808 18:33:16.625923 22744 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.78.188:43725: connect: Connection refused (error 111)
I20260808 18:33:16.626927 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.626614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43725 (local address 127.0.0.1:51862)
0808 18:33:16.626779 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.626836 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260808 18:33:16.626971 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.626724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43725 (local address 127.0.0.1:51868)
0808 18:33:16.626875 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.626924 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260808 18:33:16.627108 22718 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:43725: Network error: Client connection negotiation failed: client connection to 127.29.78.188:43725: connect: Connection refused (error 111)
W20260808 18:33:16.627136 22801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:43725: Network error: Client connection negotiation failed: client connection to 127.29.78.188:43725: connect: Connection refused (error 111)
I20260808 18:33:16.658270 22718 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } attempt: 1
I20260808 18:33:16.659160 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.658863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43725 (local address 127.0.0.1:51872)
0808 18:33:16.659009 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.659081 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 18:33:16.659902 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.659691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43725 (local address 127.0.0.1:51884)
0808 18:33:16.659798 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.659852 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 18:33:16.660035 22718 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:43725: Network error: Client connection negotiation failed: client connection to 127.29.78.188:43725: connect: Connection refused (error 111)
I20260808 18:33:16.676388 22801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } attempt: 1
I20260808 18:33:16.677263 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.676901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43725 (local address 127.0.0.1:51894)
0808 18:33:16.677089 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.677170 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 18:33:16.678042 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.677801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43725 (local address 127.0.0.1:51898)
0808 18:33:16.677925 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.677979 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 18:33:16.678191 22801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:43725: Network error: Client connection negotiation failed: client connection to 127.29.78.188:43725: connect: Connection refused (error 111)
W20260808 18:33:16.733381 22808 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 18:33:16.733625 22808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:16.733678 22808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:16.733753 22808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 18:33:16.735190 22718 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } attempt: 2
I20260808 18:33:16.736071 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.735775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43725 (local address 127.0.0.1:51910)
0808 18:33:16.735929 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.735992 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 18:33:16.736799 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.736607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43725 (local address 127.0.0.1:51922)
0808 18:33:16.736701 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.736745 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 18:33:16.736937 22718 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:43725: Network error: Client connection negotiation failed: client connection to 127.29.78.188:43725: connect: Connection refused (error 111)
W20260808 18:33:16.738229 22808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:16.738277 22808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:16.738296 22808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:16.738354 22808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:16.738392 22808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:16.743552 22808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:41171,127.29.78.189:36349,127.29.78.188:43725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/krb5kdc/kudu_127.29.78.188.keytab
--principal=kudu/127.29.78.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.188:43725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.78.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:16.744515 22808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:16.745517 22808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:33:16.746465 22801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } attempt: 2
I20260808 18:33:16.747190 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.746906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43725 (local address 127.0.0.1:51936)
0808 18:33:16.747053 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.747117 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 18:33:16.748014 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.747743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43725 (local address 127.0.0.1:51938)
0808 18:33:16.747880 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.747941 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 18:33:16.748181 22801 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:43725: Network error: Client connection negotiation failed: client connection to 127.29.78.188:43725: connect: Connection refused (error 111)
W20260808 18:33:16.754196 22817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:16.754235 22808 server_base.cc:1034] running on GCE node
W20260808 18:33:16.754252 22816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:16.754289 22819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:16.755059 22808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:16.756081 22808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:16.757272 22808 hybrid_clock.cc:648] HybridClock initialized: now 1786213996757233 us; error 54 us; skew 500 ppm
Aug 08 18:33:16 dist-test-slave-gp6n krb5kdc[22625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213996, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:16.760040 22808 init.cc:377] Logged in from keytab as kudu/127.29.78.188@KRBTEST.COM (short username kudu)
I20260808 18:33:16.760892 22808 webserver.cc:460] Webserver started at http://127.29.78.188:38543/ using document root <none> and password file <none>
I20260808 18:33:16.761386 22808 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:16.761444 22808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:16.761605 22808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:16.763597 22808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-2/data/instance:
uuid: "0344044167954892ba8ae8e0f026d72b"
format_stamp: "Formatted at 2026-08-08 18:33:16 on dist-test-slave-gp6n"
server_key: "5984cac1fb8a2a0fe36755ec69d6ccb2"
server_key_iv: "ec57ce90637558f26e885941474e0fc2"
server_key_version: "encryptionkey@0"
I20260808 18:33:16.764173 22808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-2/wal/instance:
uuid: "0344044167954892ba8ae8e0f026d72b"
format_stamp: "Formatted at 2026-08-08 18:33:16 on dist-test-slave-gp6n"
server_key: "5984cac1fb8a2a0fe36755ec69d6ccb2"
server_key_iv: "ec57ce90637558f26e885941474e0fc2"
server_key_version: "encryptionkey@0"
I20260808 18:33:16.767747 22808 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:33:16.769987 22827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:16.771246 22808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:33:16.771344 22808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-2/data,/tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-2/wal
uuid: "0344044167954892ba8ae8e0f026d72b"
format_stamp: "Formatted at 2026-08-08 18:33:16 on dist-test-slave-gp6n"
server_key: "5984cac1fb8a2a0fe36755ec69d6ccb2"
server_key_iv: "ec57ce90637558f26e885941474e0fc2"
server_key_version: "encryptionkey@0"
I20260808 18:33:16.771425 22808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:16.782387 22808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:16.784837 22808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:16.784986 22808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:16.791841 22808 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.188:43725
I20260808 18:33:16.792096 22887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.188:43725 every 8 connection(s)
I20260808 18:33:16.792783 22808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-2/data/info.pb
I20260808 18:33:16.795485 22888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:16.798861 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0kHl76/build/debug/bin/kudu as pid 22808
I20260808 18:33:16.799130 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0kHl76/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786213708769829-30010-0/minicluster-data/master-2/wal/instance
I20260808 18:33:16.799467 30010 external_mini_cluster.cc:1428] Setting key 73aee0ebd1a00025c94d7fc643fce698
I20260808 18:33:16.799284 22888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41171 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 18:33:16 dist-test-slave-gp6n krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213996, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
Aug 08 18:33:16 dist-test-slave-gp6n krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
I20260808 18:33:16.813951 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.801281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36118 (local address 127.29.78.190:41171)
0808 18:33:16.801408 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:16.801414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:16.801793 (+ 379us) server_negotiation.cc:407] Connection header received
0808 18:33:16.802488 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:16.802492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:16.802544 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:16.802640 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:16.803897 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.804682 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.806184 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.806556 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.806656 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.809705 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:16.809709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:16.809713 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:16.809760 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:16.811527 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.812096 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.812102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.812107 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.812201 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.812664 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.812667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.812668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.812895 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:16.813001 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:16.813660 (+ 659us) server_negotiation.cc:299] Negotiation successful
0808 18:33:16.813784 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 18:33:16.814863 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.801156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:41171 (local address 127.0.0.1:36118)
0808 18:33:16.801621 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.801673 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:16.802239 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:16.802722 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:16.802728 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:16.803272 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:16.803729 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:16.803755 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.804797 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.804800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:16.806033 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:16.806045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.806715 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.806718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:16.806892 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.807700 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:16.807726 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:16.809532 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:16.811648 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:16.811657 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:16.811664 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:16.811988 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:16.812323 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:16.812326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:16.812328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:16.812554 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:16.813058 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:16.813063 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:16.813300 (+ 237us) client_negotiation.cc:769] Sending connection context
0808 18:33:16.813612 (+ 312us) client_negotiation.cc:240] Negotiation successful
0808 18:33:16.813954 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":29,"spinlock_wait_cycles":896,"thread_start_us":127,"threads_started":1}
I20260808 18:33:16.814867 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.800914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51940 (local address 127.29.78.188:43725)
0808 18:33:16.801419 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:16.801438 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:16.801487 (+ 49us) server_negotiation.cc:407] Connection header received
0808 18:33:16.802222 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:16.802239 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:16.802520 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:16.802691 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:16.803768 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.804986 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.806206 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.806781 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.806885 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.810161 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:16.810187 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:16.810202 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:16.810231 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:16.812077 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.812580 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.812584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.812585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.812685 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.812969 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.812982 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.812983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.813286 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:16.813427 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:16.813885 (+ 458us) server_negotiation.cc:299] Negotiation successful
0808 18:33:16.814035 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":346,"thread_start_us":163,"threads_started":1}
I20260808 18:33:16.816179 22888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 18:33:16 dist-test-slave-gp6n krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213996, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
Aug 08 18:33:16 dist-test-slave-gp6n krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260808 18:33:16.825879 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.816704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:36349 (local address 127.0.0.1:54714)
0808 18:33:16.816856 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.816869 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:16.816937 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:16.817462 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:16.817465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:16.817676 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:16.817782 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:16.817788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.818746 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.818750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:16.819394 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:16.819403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.819976 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.819982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:16.820087 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.820731 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:16.820743 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:16.822043 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:16.824313 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:16.824316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:16.824317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:16.824508 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:16.824825 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:16.824828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:16.824830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:16.824919 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:16.825398 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:16.825402 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:16.825493 (+ 91us) client_negotiation.cc:769] Sending connection context
0808 18:33:16.825574 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 18:33:16.825717 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260808 18:33:16.825979 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.816800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54714 (local address 127.29.78.189:36349)
0808 18:33:16.817100 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:16.817104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:16.817118 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:33:16.817157 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:16.817159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:16.817230 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:16.817323 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:16.817920 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.818639 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.819535 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.819861 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.819941 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.822201 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:16.822204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:16.822206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:16.822253 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:16.824184 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.824614 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.824621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.824623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.824728 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.825029 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.825032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.825035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.825301 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:16.825403 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:16.825661 (+ 258us) server_negotiation.cc:299] Negotiation successful
0808 18:33:16.825802 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260808 18:33:16.826733 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.817300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36130 (local address 127.29.78.190:41171)
0808 18:33:16.817422 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:16.817428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:16.817441 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:33:16.817474 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:16.817477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:16.817521 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:16.817619 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:16.818239 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.818897 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.819690 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.819928 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.820049 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.822845 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:16.822849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:16.822852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:16.822881 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:16.824537 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.825024 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.825028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.825030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.825144 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.825437 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.825440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.825442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.825705 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:16.825930 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:16.826389 (+ 459us) server_negotiation.cc:299] Negotiation successful
0808 18:33:16.826554 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 18:33:16.826769 22888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 18:33:16 dist-test-slave-gp6n krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213996, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
Aug 08 18:33:16 dist-test-slave-gp6n krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
I20260808 18:33:16.834663 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.827249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43725 (local address 127.0.0.1:51954)
0808 18:33:16.827442 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.827454 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:16.827541 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:16.827769 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:16.827771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:16.827928 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:16.828017 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:16.828023 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.828730 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.828733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:16.829223 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:16.829232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.829712 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.829715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:16.829807 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.830241 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:16.830254 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:16.831594 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:16.833226 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:16.833232 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:16.833234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:16.833418 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:16.833692 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:16.833696 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:16.833698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:16.833779 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:16.834130 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:16.834134 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:16.834211 (+ 77us) client_negotiation.cc:769] Sending connection context
0808 18:33:16.834316 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 18:33:16.834481 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":1}
I20260808 18:33:16.834674 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.827336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51954 (local address 127.29.78.188:43725)
0808 18:33:16.827417 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:16.827421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:16.827494 (+ 73us) server_negotiation.cc:407] Connection header received
0808 18:33:16.827635 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:16.827638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:16.827679 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:16.827758 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:16.828142 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.828615 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.829371 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.829609 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.829715 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.831728 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:16.831730 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:16.831733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:16.831752 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:16.833125 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.833515 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.833518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.833521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.833601 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.833872 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.833875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.833877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.834029 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:16.834130 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:16.834373 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 18:33:16.834537 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":31}
I20260808 18:33:16.836119 22718 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } attempt: 3
I20260808 18:33:16.837275 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.827949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54724 (local address 127.29.78.189:36349)
0808 18:33:16.828060 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:16.828067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:16.828081 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:33:16.828212 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:16.828215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:16.828258 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:16.828358 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:16.828997 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.829603 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.830443 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.830689 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.830810 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.833428 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:16.833432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:16.833435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:16.833461 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:16.835384 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.835837 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.835841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.835843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.835935 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.836239 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.836245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.836247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.836496 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:16.836685 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:16.836985 (+ 300us) server_negotiation.cc:299] Negotiation successful
0808 18:33:16.837107 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 18:33:16.838759 22888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0344044167954892ba8ae8e0f026d72b: Bootstrap starting.
I20260808 18:33:16.839352 22801 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } attempt: 3
I20260808 18:33:16.841075 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:33:16.841703 22888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0344044167954892ba8ae8e0f026d72b: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:16.842646 22888 log.cc:824] T 00000000000000000000000000000000 P 0344044167954892ba8ae8e0f026d72b: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:16.842309 22689 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "855228b4b299464eb7d9b88230cd55d3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36130
Aug 08 18:33:16 dist-test-slave-gp6n krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213996, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
I20260808 18:33:16.845028 22888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0344044167954892ba8ae8e0f026d72b: No bootstrap required, opened a new log
Aug 08 18:33:16 dist-test-slave-gp6n krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786213996, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
I20260808 18:33:16.849264 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.836525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43725 (local address 127.0.0.1:51958)
0808 18:33:16.838555 (+ 2030us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.838570 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:16.838648 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:16.839010 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:16.839013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:16.839172 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:16.839253 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:16.839258 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.840280 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.840284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:16.840964 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:16.840977 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.841532 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.841536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:16.841642 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.843854 (+ 2212us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:16.843872 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:16.845501 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:16.847452 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:16.847455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:16.847457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:16.847720 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:16.848075 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:16.848093 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:16.848097 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:16.848194 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:16.848713 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:16.848718 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:16.848841 (+ 123us) client_negotiation.cc:769] Sending connection context
0808 18:33:16.848930 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 18:33:16.849072 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1946}
I20260808 18:33:16.850014 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.836568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51958 (local address 127.29.78.188:43725)
0808 18:33:16.838776 (+ 2208us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:16.838779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:16.838788 (+ 9us) server_negotiation.cc:407] Connection header received
0808 18:33:16.838812 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:16.838814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:16.838878 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:16.839412 (+ 534us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:16.839500 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.840173 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.841084 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.841400 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.841508 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.845640 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:16.845644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:16.845646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:16.845689 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:16.847332 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.847826 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.847832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.847835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.847921 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.848298 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.848304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.848305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.848586 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:16.849637 (+ 1051us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:16.849720 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 18:33:16.849843 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2153}
I20260808 18:33:16.850116 22888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0344044167954892ba8ae8e0f026d72b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855228b4b299464eb7d9b88230cd55d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41171 } } peers { permanent_uuid: "4c92a480dd42460eb17cec101e5e2dfb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } } peers { permanent_uuid: "0344044167954892ba8ae8e0f026d72b" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } }
I20260808 18:33:16.850376 22888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0344044167954892ba8ae8e0f026d72b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0344044167954892ba8ae8e0f026d72b, State: Initialized, Role: FOLLOWER
I20260808 18:33:16.850750 22888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0344044167954892ba8ae8e0f026d72b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855228b4b299464eb7d9b88230cd55d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41171 } } peers { permanent_uuid: "4c92a480dd42460eb17cec101e5e2dfb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } } peers { permanent_uuid: "0344044167954892ba8ae8e0f026d72b" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } }
I20260808 18:33:16.851423 22896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0344044167954892ba8ae8e0f026d72b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855228b4b299464eb7d9b88230cd55d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41171 } } peers { permanent_uuid: "4c92a480dd42460eb17cec101e5e2dfb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } } peers { permanent_uuid: "0344044167954892ba8ae8e0f026d72b" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } } }
I20260808 18:33:16.851717 22896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0344044167954892ba8ae8e0f026d72b [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:33:16.851883 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.839708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:43725 (local address 127.0.0.1:51962)
0808 18:33:16.839861 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:16.839874 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:16.839938 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:16.842842 (+ 2904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:16.842845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:16.843050 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:16.843151 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:16.843157 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.844110 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.844115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:16.844768 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:16.844778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.845726 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.845730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:16.845815 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.846356 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:16.846370 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:16.848120 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:16.850237 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:16.850243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:16.850247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:16.850553 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:16.850854 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:16.850857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:16.850859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:16.850960 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:16.851421 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:16.851424 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:16.851515 (+ 91us) client_negotiation.cc:769] Sending connection context
0808 18:33:16.851596 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 18:33:16.851724 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260808 18:33:16.852205 22888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0344044167954892ba8ae8e0f026d72b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:16.852546 22894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.839839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51962 (local address 127.29.78.188:43725)
0808 18:33:16.842595 (+ 2756us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:16.842603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:16.842616 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:33:16.842646 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:16.842649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:16.842686 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:16.842815 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:16.843269 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.843994 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.845165 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.845519 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.845617 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.848265 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:16.848269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:16.848273 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:16.848299 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:16.850116 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.850661 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.850665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.850666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.850752 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.851075 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.851082 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.851086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.851288 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:16.852183 (+ 895us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:16.852262 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 18:33:16.852382 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2691,"thread_start_us":120,"threads_started":1}
I20260808 18:33:16.853256 22718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3: Bootstrap starting.
I20260808 18:33:16.855594 22801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c92a480dd42460eb17cec101e5e2dfb: Bootstrap starting.
I20260808 18:33:16.856035 22718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:16.856874 22718 log.cc:824] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3: Log is configured to *not* fsync() on all Append() calls
W20260808 18:33:16.857483 22908 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0344044167954892ba8ae8e0f026d72b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:33:16.857534 22908 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:33:16.858052 22801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c92a480dd42460eb17cec101e5e2dfb: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:16.859280 22718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3: No bootstrap required, opened a new log
I20260808 18:33:16.859436 22801 log.cc:824] T 00000000000000000000000000000000 P 4c92a480dd42460eb17cec101e5e2dfb: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:16.855935 22808 master_runner.cc:186] Error getting master registration for 127.29.78.190:41171: OK, instance_id { permanent_uuid: "855228b4b299464eb7d9b88230cd55d3" instance_seqno: 1786213996366500 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 18:33:16.859968 22808 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 18:33:16.862381 22718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855228b4b299464eb7d9b88230cd55d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41171 } } peers { permanent_uuid: "4c92a480dd42460eb17cec101e5e2dfb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } } peers { permanent_uuid: "0344044167954892ba8ae8e0f026d72b" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } }
I20260808 18:33:16.862638 22718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 855228b4b299464eb7d9b88230cd55d3, State: Initialized, Role: FOLLOWER
I20260808 18:33:16.862968 22718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855228b4b299464eb7d9b88230cd55d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41171 } } peers { permanent_uuid: "4c92a480dd42460eb17cec101e5e2dfb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } } peers { permanent_uuid: "0344044167954892ba8ae8e0f026d72b" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } }
I20260808 18:33:16.863456 22912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855228b4b299464eb7d9b88230cd55d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41171 } } peers { permanent_uuid: "4c92a480dd42460eb17cec101e5e2dfb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } } peers { permanent_uuid: "0344044167954892ba8ae8e0f026d72b" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } } }
I20260808 18:33:16.863649 22912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:33:16.864068 22801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c92a480dd42460eb17cec101e5e2dfb: No bootstrap required, opened a new log
I20260808 18:33:16.864238 22718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:16.864382 22689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 18:33:16.864524 22689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:16.865669 22689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855228b4b299464eb7d9b88230cd55d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41171 } } peers { permanent_uuid: "4c92a480dd42460eb17cec101e5e2dfb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } } peers { permanent_uuid: "0344044167954892ba8ae8e0f026d72b" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } }
I20260808 18:33:16.867164 22856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "855228b4b299464eb7d9b88230cd55d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0344044167954892ba8ae8e0f026d72b"
I20260808 18:33:16.866632 22689 leader_election.cc:288] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3 [CANDIDATE]: Term 1 election: Requested vote from peers 4c92a480dd42460eb17cec101e5e2dfb (127.29.78.189:36349), 0344044167954892ba8ae8e0f026d72b (127.29.78.188:43725)
I20260808 18:33:16.867403 22856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0344044167954892ba8ae8e0f026d72b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:16.869220 22856 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0344044167954892ba8ae8e0f026d72b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 855228b4b299464eb7d9b88230cd55d3 in term 1.
I20260808 18:33:16.870550 22801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c92a480dd42460eb17cec101e5e2dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855228b4b299464eb7d9b88230cd55d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41171 } } peers { permanent_uuid: "4c92a480dd42460eb17cec101e5e2dfb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } } peers { permanent_uuid: "0344044167954892ba8ae8e0f026d72b" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } }
I20260808 18:33:16.870767 22801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c92a480dd42460eb17cec101e5e2dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c92a480dd42460eb17cec101e5e2dfb, State: Initialized, Role: FOLLOWER
W20260808 18:33:16.871106 22928 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260808 18:33:16.871101 22801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c92a480dd42460eb17cec101e5e2dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855228b4b299464eb7d9b88230cd55d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41171 } } peers { permanent_uuid: "4c92a480dd42460eb17cec101e5e2dfb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } } peers { permanent_uuid: "0344044167954892ba8ae8e0f026d72b" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } }
W20260808 18:33:16.871248 22928 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:33:16.871331 22658 leader_election.cc:302] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0344044167954892ba8ae8e0f026d72b, 855228b4b299464eb7d9b88230cd55d3; no voters:
I20260808 18:33:16.871582 22912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:16.871776 22912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3 [term 1 LEADER]: Becoming Leader. State: Replica: 855228b4b299464eb7d9b88230cd55d3, State: Running, Role: LEADER
I20260808 18:33:16.872073 22912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855228b4b299464eb7d9b88230cd55d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41171 } } peers { permanent_uuid: "4c92a480dd42460eb17cec101e5e2dfb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } } peers { permanent_uuid: "0344044167954892ba8ae8e0f026d72b" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } }
I20260808 18:33:16.872457 22801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c92a480dd42460eb17cec101e5e2dfb [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:16.869607 22640 master_runner.cc:186] Error getting master registration for 127.29.78.189:36349: OK, instance_id { permanent_uuid: "4c92a480dd42460eb17cec101e5e2dfb" instance_seqno: 1786213996578320 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 18:33:16.873521 22640 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 18:33:16.873632 22773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "855228b4b299464eb7d9b88230cd55d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4c92a480dd42460eb17cec101e5e2dfb"
I20260808 18:33:16.873840 22773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c92a480dd42460eb17cec101e5e2dfb [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:16.873843 22912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 855228b4b299464eb7d9b88230cd55d3. Latest consensus state: current_term: 1 leader_uuid: "855228b4b299464eb7d9b88230cd55d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855228b4b299464eb7d9b88230cd55d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41171 } } peers { permanent_uuid: "4c92a480dd42460eb17cec101e5e2dfb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } } peers { permanent_uuid: "0344044167954892ba8ae8e0f026d72b" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } } }
I20260808 18:33:16.873986 22912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:16.875190 22773 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c92a480dd42460eb17cec101e5e2dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 855228b4b299464eb7d9b88230cd55d3 in term 1.
W20260808 18:33:16.878383 22949 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c92a480dd42460eb17cec101e5e2dfb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:33:16.878553 22949 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:33:16.879726 22942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:16.880658 22942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:16.889601 22934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c92a480dd42460eb17cec101e5e2dfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855228b4b299464eb7d9b88230cd55d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41171 } } peers { permanent_uuid: "4c92a480dd42460eb17cec101e5e2dfb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } } peers { permanent_uuid: "0344044167954892ba8ae8e0f026d72b" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } } }
I20260808 18:33:16.889750 22934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c92a480dd42460eb17cec101e5e2dfb [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:33:16.892110 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.878845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36134 (local address 127.29.78.190:41171)
0808 18:33:16.878955 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:16.878960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:16.883004 (+ 4044us) server_negotiation.cc:407] Connection header received
0808 18:33:16.883156 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:16.883159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:16.883201 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:16.883249 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:16.884249 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.885035 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.886229 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.886595 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.886668 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.888498 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:16.888502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:16.888506 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:16.888532 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:16.890095 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.890745 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.890749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.890751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.890850 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.891140 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.891143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.891145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.891349 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:16.891452 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:16.891820 (+ 368us) server_negotiation.cc:299] Negotiation successful
0808 18:33:16.891947 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 18:33:16.896168 22856 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0344044167954892ba8ae8e0f026d72b [term 1 FOLLOWER]: Refusing update from remote peer 855228b4b299464eb7d9b88230cd55d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:16.897007 22773 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4c92a480dd42460eb17cec101e5e2dfb [term 1 FOLLOWER]: Refusing update from remote peer 855228b4b299464eb7d9b88230cd55d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:16.897902 22912 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c92a480dd42460eb17cec101e5e2dfb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:33:16.901309 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.879074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51968 (local address 127.29.78.188:43725)
0808 18:33:16.879208 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:16.879212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:16.882698 (+ 3486us) server_negotiation.cc:407] Connection header received
0808 18:33:16.882845 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:16.882848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:16.882903 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:16.882996 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:16.891929 (+ 8933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.892667 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.894460 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.894770 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.894848 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.897623 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:16.897628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:16.897634 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:16.897663 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:16.899425 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.899911 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.899915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.899916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.900017 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.900328 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.900331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.900333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.900540 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:16.900640 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:16.901018 (+ 378us) server_negotiation.cc:299] Negotiation successful
0808 18:33:16.901147 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 18:33:16.898373 22912 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0344044167954892ba8ae8e0f026d72b" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260808 18:33:16.903568 22934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c92a480dd42460eb17cec101e5e2dfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 855228b4b299464eb7d9b88230cd55d3. Latest consensus state: current_term: 1 leader_uuid: "855228b4b299464eb7d9b88230cd55d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855228b4b299464eb7d9b88230cd55d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41171 } } peers { permanent_uuid: "4c92a480dd42460eb17cec101e5e2dfb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } } peers { permanent_uuid: "0344044167954892ba8ae8e0f026d72b" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } } }
I20260808 18:33:16.904021 22934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c92a480dd42460eb17cec101e5e2dfb [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:33:16.906738 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:16.879496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.29.78.189:36349)
0808 18:33:16.879604 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:16.879608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:16.884149 (+ 4541us) server_negotiation.cc:407] Connection header received
0808 18:33:16.884320 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:16.884323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:16.884370 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:16.884419 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:16.885393 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.886160 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.887339 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:16.887695 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:16.887779 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:16.903123 (+ 15344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:16.903127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:16.903130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:16.903161 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:16.904773 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.905298 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.905301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.905303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.905391 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:16.905746 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:16.905749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:16.905751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:16.905959 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:16.906068 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:16.906449 (+ 381us) server_negotiation.cc:299] Negotiation successful
0808 18:33:16.906579 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 18:33:16.907279 22896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0344044167954892ba8ae8e0f026d72b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 855228b4b299464eb7d9b88230cd55d3. Latest consensus state: current_term: 1 leader_uuid: "855228b4b299464eb7d9b88230cd55d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855228b4b299464eb7d9b88230cd55d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41171 } } peers { permanent_uuid: "4c92a480dd42460eb17cec101e5e2dfb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } } peers { permanent_uuid: "0344044167954892ba8ae8e0f026d72b" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } } }
I20260808 18:33:16.907421 22896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0344044167954892ba8ae8e0f026d72b [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:33:16.913187 22912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "855228b4b299464eb7d9b88230cd55d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855228b4b299464eb7d9b88230cd55d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41171 } } peers { permanent_uuid: "4c92a480dd42460eb17cec101e5e2dfb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } } peers { permanent_uuid: "0344044167954892ba8ae8e0f026d72b" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } } }
I20260808 18:33:16.913333 22912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:16.913715 22934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c92a480dd42460eb17cec101e5e2dfb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "855228b4b299464eb7d9b88230cd55d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855228b4b299464eb7d9b88230cd55d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41171 } } peers { permanent_uuid: "4c92a480dd42460eb17cec101e5e2dfb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } } peers { permanent_uuid: "0344044167954892ba8ae8e0f026d72b" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } } }
I20260808 18:33:16.913838 22934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c92a480dd42460eb17cec101e5e2dfb [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:33:16.914690 22955 mvcc.cc:205] Tried to move back new op lower bound from 7316332531262889984 to 7316332531191353344. Current Snapshot: MvccSnapshot[applied={T|T < 7316332531262889984}]
I20260808 18:33:16.915745 22912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "855228b4b299464eb7d9b88230cd55d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855228b4b299464eb7d9b88230cd55d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41171 } } peers { permanent_uuid: "4c92a480dd42460eb17cec101e5e2dfb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } } peers { permanent_uuid: "0344044167954892ba8ae8e0f026d72b" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } } }
I20260808 18:33:16.915860 22912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:16.916987 22896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0344044167954892ba8ae8e0f026d72b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "855228b4b299464eb7d9b88230cd55d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855228b4b299464eb7d9b88230cd55d3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41171 } } peers { permanent_uuid: "4c92a480dd42460eb17cec101e5e2dfb" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 36349 } } peers { permanent_uuid: "0344044167954892ba8ae8e0f026d72b" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 43725 } } }
I20260808 18:33:16.917104 22896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0344044167954892ba8ae8e0f026d72b [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:33:16.917790 22957 mvcc.cc:205] Tried to move back new op lower bound from 7316332531262889984 to 7316332531191353344. Current Snapshot: MvccSnapshot[applied={T|T < 7316332531262889984}]
I20260808 18:33:16.919708 22942 catalog_manager.cc:1347] Generated new cluster ID: eda410d24c084328923ba974753f2332
I20260808 18:33:16.919796 22942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:16.938696 22942 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:16.939958 22942 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:16.958464 22942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 855228b4b299464eb7d9b88230cd55d3: Generated new TSK 0
I20260808 18:33:16.959385 22942 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 18:33:16.992365 22722 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:16.989486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36148 (local address 127.29.78.190:41171)
0808 18:33:16.989620 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:16.989625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:16.991235 (+ 1610us) server_negotiation.cc:407] Connection header received
0808 18:33:16.991453 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:16.991457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:16.991510 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:16.991565 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:16.992207 (+ 642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36148: BlockingRecv error: recv got EOF from 127.0.0.1:36148 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260808 18:33:16.993709 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:16.990227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54748 (local address 127.29.78.189:36349)
0808 18:33:16.990358 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:16.990362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:16.992844 (+ 2482us) server_negotiation.cc:407] Connection header received
0808 18:33:16.993012 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:16.993015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:16.993061 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:16.993114 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:16.993596 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54748: BlockingRecv error: recv got EOF from 127.0.0.1:54748 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260808 18:33:16.995204 22891 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:16.990952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51980 (local address 127.29.78.188:43725)
0808 18:33:16.991079 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:16.991083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:16.994075 (+ 2992us) server_negotiation.cc:407] Connection header received
0808 18:33:16.994467 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:16.994471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:16.994524 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:16.994575 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:16.995089 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51980: BlockingRecv error: recv got EOF from 127.0.0.1:51980 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260808 18:33:16.999382 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:16.997979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54754 (local address 127.29.78.189:36349)
0808 18:33:16.998084 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:16.998088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:16.998455 (+ 367us) server_negotiation.cc:407] Connection header received
0808 18:33:16.998678 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:16.998681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:16.998753 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:16.998801 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:16.999271 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54754: BlockingRecv error: recv got EOF from 127.0.0.1:54754 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260808 18:33:17.004289 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:17.002880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54758 (local address 127.29.78.189:36349)
0808 18:33:17.002977 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:17.002981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:17.003464 (+ 483us) server_negotiation.cc:407] Connection header received
0808 18:33:17.003636 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:17.003639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:17.003684 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:17.003732 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:17.004196 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54758: BlockingRecv error: recv got EOF from 127.0.0.1:54758 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260808 18:33:17.009585 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:17.008345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54760 (local address 127.29.78.189:36349)
0808 18:33:17.008455 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:17.008462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:17.008744 (+ 282us) server_negotiation.cc:407] Connection header received
0808 18:33:17.008910 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:17.008913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:17.008956 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:17.009003 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:17.009489 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54760: BlockingRecv error: recv got EOF from 127.0.0.1:54760 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260808 18:33:17.017249 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:17.015690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54764 (local address 127.29.78.189:36349)
0808 18:33:17.015810 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:17.015814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:17.016305 (+ 491us) server_negotiation.cc:407] Connection header received
0808 18:33:17.016512 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:17.016515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:17.016560 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:17.016607 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:17.017155 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54764: BlockingRecv error: recv got EOF from 127.0.0.1:54764 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260808 18:33:17.029385 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:17.027686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54772 (local address 127.29.78.189:36349)
0808 18:33:17.027803 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:17.027808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:17.028371 (+ 563us) server_negotiation.cc:407] Connection header received
0808 18:33:17.028584 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:17.028588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:17.028639 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:17.028691 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:17.029239 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54772: BlockingRecv error: recv got EOF from 127.0.0.1:54772 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260808 18:33:17.050269 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:17.048315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54784 (local address 127.29.78.189:36349)
0808 18:33:17.048468 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:17.048473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:17.049115 (+ 642us) server_negotiation.cc:407] Connection header received
0808 18:33:17.049389 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:17.049393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:17.049455 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:17.049517 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:17.050146 (+ 629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54784: BlockingRecv error: recv got EOF from 127.0.0.1:54784 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260808 18:33:17.086313 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:17.084966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54786 (local address 127.29.78.189:36349)
0808 18:33:17.085100 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:17.085104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:17.085388 (+ 284us) server_negotiation.cc:407] Connection header received
0808 18:33:17.085565 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:17.085571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:17.085628 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:17.085720 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:17.086185 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54786: BlockingRecv error: recv got EOF from 127.0.0.1:54786 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 18:33:17.154096 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:17.152709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54788 (local address 127.29.78.189:36349)
0808 18:33:17.152884 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:17.152889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:17.153024 (+ 135us) server_negotiation.cc:407] Connection header received
0808 18:33:17.153278 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:17.153286 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:17.153367 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:17.153466 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:17.153953 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54788: BlockingRecv error: recv got EOF from 127.0.0.1:54788 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260808 18:33:17.285660 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:17.284390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54794 (local address 127.29.78.189:36349)
0808 18:33:17.284564 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:17.284569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:17.284714 (+ 145us) server_negotiation.cc:407] Connection header received
0808 18:33:17.284899 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:17.284903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:17.284958 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:17.285061 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:17.285550 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54794: BlockingRecv error: recv got EOF from 127.0.0.1:54794 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260808 18:33:17.545132 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:17.543994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54808 (local address 127.29.78.189:36349)
0808 18:33:17.544162 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:17.544167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:17.544184 (+ 17us) server_negotiation.cc:407] Connection header received
0808 18:33:17.544330 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:17.544333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:17.544389 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:17.544486 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:17.544987 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54808: BlockingRecv error: recv got EOF from 127.0.0.1:54808 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 18:33:17.858929 22908 catalog_manager.cc:1259] Loaded cluster ID: eda410d24c084328923ba974753f2332
I20260808 18:33:17.859023 22908 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0344044167954892ba8ae8e0f026d72b: loading cluster ID for follower catalog manager: success
I20260808 18:33:17.860723 22908 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0344044167954892ba8ae8e0f026d72b: acquiring CA information for follower catalog manager: success
I20260808 18:33:17.861914 22908 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0344044167954892ba8ae8e0f026d72b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 18:33:17.879809 22949 catalog_manager.cc:1259] Loaded cluster ID: eda410d24c084328923ba974753f2332
I20260808 18:33:17.879920 22949 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4c92a480dd42460eb17cec101e5e2dfb: loading cluster ID for follower catalog manager: success
I20260808 18:33:17.881562 22949 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4c92a480dd42460eb17cec101e5e2dfb: acquiring CA information for follower catalog manager: success
I20260808 18:33:17.882745 22949 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4c92a480dd42460eb17cec101e5e2dfb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 18:33:18.064000 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:18.059497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54816 (local address 127.29.78.189:36349)
0808 18:33:18.059798 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:18.059803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:18.059931 (+ 128us) server_negotiation.cc:407] Connection header received
0808 18:33:18.060123 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:18.060126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:18.060267 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:18.060391 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:18.060695 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:18.061391 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:18.062441 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:18.062683 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:18.062796 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:18.063131 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:18.063561 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:18.063701 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:18.063774 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 18:33:18.063832 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":90,"threads_started":1}
I20260808 18:33:18.070513 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:18.067171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51986 (local address 127.29.78.188:43725)
0808 18:33:18.067414 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:18.067418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:18.067431 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:33:18.067562 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:18.067564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:18.067670 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:18.067762 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:18.067938 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:18.068433 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:18.069199 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:18.069418 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:18.069494 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:18.069725 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:18.070122 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:18.070211 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:18.070262 (+ 51us) server_negotiation.cc:299] Negotiation successful
0808 18:33:18.070306 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":70,"threads_started":1}
I20260808 18:33:18.077581 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:18.073855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54824 (local address 127.29.78.189:36349)
0808 18:33:18.073965 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:18.073969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:18.074195 (+ 226us) server_negotiation.cc:407] Connection header received
0808 18:33:18.074369 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:18.074374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:18.074454 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:18.074520 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:18.074913 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:18.075423 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:18.076535 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:18.076747 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:18.076849 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:18.077136 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:18.077214 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:18.077292 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:18.077395 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 18:33:18.077459 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 18:33:18.077620 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:18.073661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52002 (local address 127.29.78.188:43725)
0808 18:33:18.073779 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:18.073783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:18.074142 (+ 359us) server_negotiation.cc:407] Connection header received
0808 18:33:18.074282 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:18.074285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:18.074354 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:18.074436 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:18.074665 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:18.075334 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:18.076476 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:18.076774 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:18.076841 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:18.077187 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:18.077255 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:18.077320 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:18.077474 (+ 154us) server_negotiation.cc:299] Negotiation successful
0808 18:33:18.077519 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 18:33:18.079164 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:18.073790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36164 (local address 127.29.78.190:41171)
0808 18:33:18.074494 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:18.074498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:18.074511 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:33:18.074598 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:18.074604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:18.074776 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:18.074913 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:18.075795 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:18.076560 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:18.077674 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:18.078029 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:18.078130 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:18.078560 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:18.078849 (+ 289us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:18.078925 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:18.079011 (+ 86us) server_negotiation.cc:299] Negotiation successful
0808 18:33:18.079059 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":58,"threads_started":1}
I20260808 18:33:18.082005 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 22640
I20260808 18:33:18.093477 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 22726
I20260808 18:33:18.100116 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0kHl76/build/debug/bin/kudu with pid 22808
2026-08-08T18:33:18Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3200 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6404 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (289317 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST